Communication management device, image communication system, communication management method, and recording medium

ABSTRACT

A communication management device includes circuitry that: receives captured image data of a captured image obtained by an image capturing device and time-series information representing a date and time associated with predetermined area information, the predetermined area information representing a predetermined area and is transmitted by a communication terminal that displays a predetermined area image of the predetermined area; and transmits, to the communication terminal, converted predetermined area image data obtained through perspective projection conversion based on the received captured image data and the received time-series information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application Nos. 2021-048088, filedon Mar. 23, 2021, and 2022-030569, filed on Mar. 1, 2022, in the JapanPatent Office, the entire disclosure of which is hereby incorporated byreference herein.

BACKGROUND Technical Field

The present disclosure relates to a communication management device, animage communication system, a communication management method, andcarrier means.

Description of the Related Art

Videoconference systems that enable a remote conference (meeting) amongremote sites via a communication network such as the Internet are now inwidespread use. In such videoconference systems, a communicationterminal of the videoconference systems is provided in a conference room(meeting room) where attendees of one party in a remote conference arepresent. This communication terminal captures an image of the conferenceroom including the attendees of the conference and collects sound suchas speeches made by the attendees, converts the image and sound intodigital data, and transmits the digital data to a counterpartcommunication terminal. The image and sound are then respectively outputon a display and from a speaker provided in a conference room of thecounterpart. In this manner, a video call can be performed, and thus aconference can be carried out among remote sites in a state close to areal conference.

There is also disclosed a camera system in which a plurality of cameraunits including a camera unit capable of using a 360-degreeomnidirectional camera are installed at an event site. The camera systemdistributes videos in real time. In response to a user selecting a videofrom among the distributed videos, the camera system transmits selectioninformation regarding the selected video to a server, so that the serveredits the video on the basis of the selection information. In thismanner, the camera system sells the edited video to the user.

However, the technique of the related art has an issue in that it isdifficult for a user to grasp when and from which viewpoint the userviewed a captured image when the user desires to view the captured imageagain after an elapse of a certain period from the distribution in asystem that captures an image and distributes the captured imageconcurrently.

SUMMARY

Example embodiments include a communication management device includingcircuitry that receives captured image data of a captured image obtainedby an image capturing device and time-series information representing adate and time associated with predetermined area information, thepredetermined area information representing a predetermined area and istransmitted by a communication terminal that displays a predeterminedarea image of the predetermined area. The circuitry transmits, to thecommunication terminal, converted predetermined area image data obtainedthrough perspective projection conversion based on the received capturedimage data and the received time-series information.

Example embodiments include an image communication system including animage capturing device, a communication terminal, and the communicationmanagement device.

Example embodiments include a communication management method including:receiving captured image data of a captured image obtained by an imagecapturing device and time-series information representing a date andtime associated with predetermined area information, the predeterminedarea information representing a predetermined area and is transmitted bya communication terminal that displays a predetermined area image of thepredetermined area; and transmitting, to the communication terminal,converted predetermined area image data obtained through perspectiveprojection conversion based on the received captured image data and thereceived time-series information.

Example embodiments include a non-transitory recording medium which,when executed by one or more processors, cause the processors to performthe communication management method.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1A is a left side view of an image capturing device;

FIG. 1B is a rear view of the image capturing device:

FIG. 1C is a plan view of the image capturing device;

FIG. 2 is an illustration of an example of how the image capturingdevice is used;

FIG. 3A is an illustration of a hemispherical image (front side)captured by the image capturing device;

FIG. 3B is an illustration of a hemispherical image (back side) capturedby the image capturing device;

FIG. 3C is an illustration of an image represented by the Mercatorprojection;

FIG. 4A is an illustration of how the Mercator image is mapped onto asphere;

FIG. 4B is an illustration of a spherical panoramic image;

FIG. 5 is a diagram illustrating positions of a virtual camera and apredetermined area in the case where the spherical panoramic image isrepresented as a three-dimensional solid sphere;

FIG. 6A is a perspective view of FIG. 5;

FIG. 6B is an illustration of a predetermined area image displayed on adisplay of a communication terminal;

FIG. 7 is a diagram illustrating a relation between predetermined areainformation and an image of a predetermined area;

FIG. 8 is a diagram illustrating a point in a three-dimensionalEuclidean space defined in spherical coordinates;

FIG. 9 is a schematic diagram illustrating an image communication systemaccording to an embodiment of the present disclosure;

FIG. 10 is a block diagram illustrating a hardware configuration of animage capturing device;

FIG. 11 is a block diagram illustrating a hardware configuration of avideoconference terminal;

FIG. 12 is a block diagram illustrating a hardware configuration of acommunication management system and a hardware configuration of apersonal computer (PC);

FIG. 13 is a block diagram illustrating a hardware configuration of asmartphone;

FIG. 14 is a block diagram illustrating a functional configuration ofpart of the image communication system;

FIG. 15 is a block diagram illustrating a functional configuration ofanother part of the image communication system;

FIG. 16A is an illustration of an image type management table;

FIG. 16B is an illustration of an image type management table;

FIG. 16C is an illustration of an image capturing device managementtable;

FIG. 16D is an illustration of a predetermined area management table;

FIG. 16E is an illustration of a session management table;

FIG. 16F is an illustration of an image type management table;

FIG. 16G is an illustration of a predetermined area management table;

FIG. 16H is an illustration of a predetermined area management tableused in the case of post-conversion recording;

FIG. 16I is an illustration of a recorded file storage table:

FIG. 17 is a sequence diagram illustrating a process of participating ina specific communication session;

FIG. 18 is an illustration of a communication session (virtualconference room) selection screen:

FIG. 19 is a sequence diagram illustrating a process of managing imagetype information;

FIG. 20A is an illustration of how a video call is performed withoutusing the image capturing device;

FIG. 20B is an illustration of how a video call is performed by usingthe image capturing device;

FIG. 21 is a sequence diagram illustrating a process of communicatingcaptured image data and audio data in a video call;

FIG. 22A is an illustration of an example of a screen displayed on adisplay at a viewing site D when images based on image data transmittedfrom the image capturing devices are displayed as they are, that is,without generating a spherical panoramic image and a predetermined areaimage;

FIG. 22B is an illustration of an example of a screen displayed on thedisplay at the viewing site D when a spherical panoramic image and apredetermined area image are generated based on image data transmittedfrom the image capturing devices;

FIG. 22C is an illustration of an example of a screen displayed on thedisplay at the viewing site D when the predetermined area image of FIG.22B is changed;

FIG. 23 is a sequence diagram illustrating a process of sharingpredetermined area information;

FIG. 24 is a flowchart illustrating a process of displaying apredetermined area image;

FIGS. 25A and 25B are diagrams presenting a method for deriving aposition of a point of gaze at another site in a predetermined areaimage of a site of interest;

FIG. 26A is a diagram illustrating definitions of angles; FIG. 26B is adiagram illustrating definitions of angle ranges;

FIGS. 27A, 27B, and 27C illustrate images displayed in a main displayarea, which are display examples of the predetermined area imageincluding display direction marks;

FIGS. 28A and 28B illustrate images displayed in the main display area,which are display examples of the predetermined area image includingpoint-of-gaze marks;

FIG. 29 illustrates an image displayed in the main display area, whichis a display example of the predetermined area image including apoint-of-gaze mark and display direction marks;

FIG. 30 is a sequence diagram illustrating another process of sharingthe predetermined area information;

FIG. 31 is a sequence diagram illustrating a process of communicatingcaptured image data and audio data in a video call:

FIG. 32 a flowchart illustrating a process of selecting live recordingor post-conversion recording and of performing live recording;

FIG. 33 is a flowchart illustrating a recording process in the case ofpost-conversion recording;

FIG. 34 illustrates an example of a file selection screen in an app forviewing a video of post-conversion recording; and

FIG. 35 illustrates an example of a movie playback screen for a video ofpost-conversion recording selected in the app.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted. Also, identical or similar referencenumerals designate identical or similar components throughout theseveral views.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Referring now to the drawings, embodiments of the present disclosure aredescribed below. As used herein, the singular forms “a,” “an,” and “the”are intended to include the plural forms as well, unless the contextclearly indicates otherwise.

First Embodiment

A first embodiment of the present disclosure is described with referenceto FIGS. 1A to 29.

Overview of Embodiment Spherical Panoramic Image Generation Method

A method for generating a spherical panoramic image is described belowwith reference to FIGS. 1A to 8.

An external view of an image capturing device 1 is described first withreference to FIGS. 1A to 1C. FIGS. 1A, 1B, and 1C are a left side view,a rear view, and a plan view of the image capturing device 1,respectively. The image capturing device 1 is a spherical imagecapturing device capable of capturing a three-dimensional (3D) sphericalpanoramic image (images from which a 3D spherical panoramic image isgenerated). Examples of the spherical image capturing device include adigital camera. The term “spherical” indicates a space of 360 degrees inall directions including upper, lower, left, and right directions.

As illustrated in FIG. 1A, the image capturing device 1 can be held by aperson with a single hand. As illustrated in FIGS. 1A, 1B, and 1C, theimage capturing device 1 includes an imaging element 103 a and animaging element 103 b respectively on a front surface side (front side)and a back surface side (back side) in an upper portion thereof. Theseimaging elements (image sensors) 103 a and 103 b are used in combinationwith optical members (for example, fisheye lenses 102 a and 102 b inFIG. 10 described below) each capable of capturing a hemispherical image(of an angle of view of 180 degrees or wider). As illustrated in FIG.1B, the image capturing device 1 includes an operation device 115 suchas a shutter button on a surface opposite to the front surface side.

How the image capturing device 1 is used is described next withreference to FIG. 2. FIG. 2 is an illustration of an example of how theimage capturing device 1 is used. As illustrated in FIG. 2, the imagecapturing device 1 is used for capturing an image of subjects or objectslocated around the user who is holding the image capturing device 1 inhis or her hand, for example. In this case, the imaging elements 103 aand 103 b illustrated in FIGS. 1A to 1C capture images of the subjectsor objects located around the user to obtain two hemispherical images.

An overview of a process of generating a spherical panoramic image fromthe images captured by the image capturing device 1 is described nextwith reference to FIGS. 3A to 4B. FIG. 3A illustrates a hemisphericalimage (front side) captured by the image capturing device 1. FIG. 3Billustrates a hemispherical image (back side) captured by the imagecapturing device 1. FIG. 3C illustrates an image represented by theMercator projection (hereinafter referred to as a “Mercator image”).FIG. 4A is an illustration of how the Mercator image is mapped onto asphere. FIG. 4B illustrates a spherical panoramic image.

As illustrated in FIG. 3A, an image captured by the imaging element 103a is a curved hemispherical image (front side) captured through thefisheye lens 102 a (described later). Also, as illustrated in FIG. 3B,an image captured by the imaging element 103 b is a curved hemisphericalimage (back side) captured through the fisheye lens 102 b (describedlater). The image capturing device 1 combines the hemispherical image(front side) and the hemispherical image (back side), which is reversedby 180 degrees from the hemispherical image (front side), to generatethe Mercator image as illustrated in FIG. 3C.

Open Graphics Library for Embedded Systems (OpenGL ES) is used, so thatthe Mercator image is attached so as to cover the sphere surface asillustrated in FIG. 4A. Consequently, the spherical panoramic imageillustrated in FIG. 4B is generated. Thus, the spherical panoramic imageis represented as an image in which the Mercator image is directedtoward the center of the sphere. OpenGL ES is a graphics library usedfor visualizing data of two-dimensions (2D) and data of three-dimensions(3D). The spherical panoramic image may be either a still image or amovie.

Because the spherical panoramic image is an image attached so as tocover the sphere surface as described above, the spherical panoramicimage may give a feeling of strangeness to a person who views the image.Accordingly, a predetermined partial area (hereinafter, referred to as a“predetermined area image”) of the spherical panoramic image isdisplayed as a less-curved planar image so as to enable a display thatreduces the feeling of strangeness given to a person. Displaying thepredetermined area image will be described with reference to FIGS. 5 to6B).

FIG. 5 is a diagram illustrating positions of a virtual camera IC and apredetermined area T in the case where the spherical image isrepresented as a three-dimensional solid sphere CS. The virtual cameraIC corresponds to a position of a point of view (viewpoint) of a userwho is viewing a spherical image CE displayed as the three-dimensionalsolid sphere CS. FIG. 6A is a perspective view of FIG. 5. FIG. 6B is anillustration of the predetermined area image displayed on a display. InFIG. 6A, the spherical image CE illustrated in FIG. 4A is represented asthe three-dimensional solid sphere CS. When the spherical image CE thusgenerated is represented as the solid sphere CS, the virtual camera ICis located inside the spherical image CE as illustrated in FIG. 5. Thepredetermined area T in the spherical image CE is an imaging area of thevirtual camera IC. Specifically, the predetermined area T is identifiedby predetermined area information indicating an imaging direction and anangle of view of the virtual camera IC in a three-dimensional virtualspace containing the spherical image CE.

The predetermined area image, which is an image of the predeterminedarea T illustrated in FIG. 6A, is displayed on a predetermined displayas an image of the imaging area of the virtual camera IC, as illustratedin FIG. 6B. The image illustrated in FIG. 6B is the predetermined areaimage represented by the predetermined area information that is set bydefault. In another example, the predetermined area information may berepresented based on an imaging area (X, Y, Z) of the virtual camera IC,i.e., the predetermined area T, instead of the coordinates of theposition of the virtual camera IC. Description is given below by usingan imaging direction (rH, rV) and an angle of view (a) of the virtualcamera IC.

A relation between the predetermined area information and the image ofthe predetermined area T is described with reference to FIG. 7. FIG. 7is a diagram illustrating a relation between the predetermined areainformation and the image of the predetermined area T. As illustrated inFIG. 7, “rH”, “rV”, and “a” denote a horizontal radian, a verticalradian, and an angle of view, respectively. The orientation of thevirtual camera IC is adjusted such that the point of gaze of the virtualcamera IC, indicated by the imaging direction (rH, rV), matches a centerpoint CP of the predetermined area T that is the imaging area of thevirtual camera IC. The predetermined area image is the image of thepredetermined area T in the spherical image CE. “f” denotes a distancefrom the virtual camera IC to the center point CP. “L” denotes adistance between the center point CP and a given vertex of thepredetermined area T (2L denotes a diagonal line). In FIG. 7, atrigonometric function commonly represented by (Equation 1) below holds.

L/f=tan(α/2)  (Equation 1)

FIG. 8 is a diagram illustrating a point in a three-dimensionalEuclidean space defined in spherical coordinates. Coordinates of theposition of the center point CP represented by a spherical polarcoordinates system are denoted by (r, 0, φ). “r”, “0”, and “φ” of thecoordinates (r, 0, φ) represent a radius vector, a polar angle, and anazimuth angle, respectively. The radius vector r is a distance from theorigin of the three-dimensional virtual space containing the sphericalpanoramic image to the center point CP. Thus, the radius vector r isequal to f. FIG. 8 illustrates the relation between these parameters.Description is given below by using the coordinates (r, 0, φ) of thecenter point CP of the imaging area of the virtual camera IC.

Overview of Image Communication System

An overview of a configuration of an image communication systemaccording to the present embodiment is described next with reference toFIG. 9. FIG. 9 is a schematic diagram illustrating a configuration ofthe image communication system according to the present embodiment.

As illustrated in FIG. 9, the image communication system according tothe present embodiment includes image capturing devices 1 a and 1 b,videoconference terminals 3 a and 3 d, displays 4 a and 4 d, acommunication management system 5, a personal computer (PC) 7, an imagecapturing device 8, and a smartphone 9. The videoconference terminals 3a and 3 d, the communication management system 5, the PC 7, and thesmartphone 9 are capable of communicating with each other via acommunication network 100 such as Internet. The communication network100 may be either a wireless network or a wired network.

Each of the image capturing devices 1 a and 1 b is a special digitalcamera that captures an image of a subject, an object, or a scenery toobtain two hemispherical images from which a spherical panoramic imageis generated, as described above. On the other hand, the image capturingdevice 8 is a general digital camera that captures an image of asubject, an object, or a scenery to obtain a general planar image.

The videoconference terminals 3 a and 3 d are terminals dedicated forvideoconferencing, and display an image of a video call on the displays4 a and 4 d via a cable such as a Universal Serial Bus (USB) cable,respectively. The videoconference terminal 3 a usually captures an imagewith a camera 312 illustrated in FIG. 11 (described later). When thevideoconference terminal 3 a is coupled by a cable to a cradle 2 a towhich the image capturing device 1 a is mounted, the image capturingdevice 1 a is preferentially used. In this manner, the videoconferenceterminal 3 a can obtain two hemispherical images from which a sphericalpanoramic image is generated. When a cable is used for coupling thevideoconference terminal 3 a and the cradle 2 a to each other, thecradle 2 a supplies power to the image capturing device 1 a and holdsthe image capturing device 1 a as well as relays communication betweenthe image capturing device 1 a and the videoconference terminal 3 a. Inthe present embodiment, the image capturing device 1 a, the cradle 2 a,the videoconference terminal 3 a, and the display 4 a are placed at thesame site, namely, a viewing site A. In addition, at the viewing site A,four users, namely, users A1, A2, A3, and A4 are participating in thevideo call. In addition, the videoconference terminal 3 d and thedisplay 4 d are placed at the same site, namely, a viewing site D. Atthe viewing site D, three users, namely, users D1, D2, and D3 areparticipating in the video call.

The communication management system 5 manages and controls communicationamong the videoconference terminals 3 a and 3 d, the PC 7, and thesmartphone 9 and manages types of image data (types such as a generalimage and a special image) to be transmitted and received among thevideoconference terminals 3 a and 3 d, the PC 7, and the smartphone 9.Thus, the communication management system 5 is also a communicationcontrol system. In the present embodiment, the special image refers to aspherical panoramic image. The communication management system 5 isinstalled at, for example, an office of a service provider that providesa video communication service. The communication management system 5 maybe configured as a single computer. Alternatively, the communicationmanagement system 5 may be configured as a plurality of computers, toeach of which one or more units (functions, means, or storages) areassigned in any manner. In the present embodiment, the communicationmanagement system 5 functions as an example of a communicationmanagement device.

The PC 7 can perform a video call when equipped with the image capturingdevice 8. In the present embodiment, the PC 7 and the image capturingdevice 8 are placed at the same site, namely, a viewing site C. At theviewing site C, one user, namely, a user C1 is participating in thevideo call.

The smartphone 9 includes a display 917 (described later), and displaysan image of the video call on the display 917. The smartphone 9 includesa complementary metal oxide semiconductor (CMOS) sensor 905 (describedlater), and usually captures an image with the CMOS sensor 905. Thesmartphone 9 is also capable of obtaining data of two hemisphericalimages captured by the image capturing device 1 b, from which aspherical panoramic image is generated, by using a short-range wirelesscommunication technology such as Near Field Communication (NFC,registered trademark hereinafter omitted), Bluetooth (registeredtrademark hereinafter omitted), or Wireless Fidelity (Wi-Fi, registeredtrademark hereinafter omitted). When such a short-range wirelesscommunication technology is used, a cradle 2 b just supplies power tothe image capturing device 1 b and holds the image capturing device 1 b.In the present embodiment, the image capturing device 1 b, the cradle 2b, and the smartphone 9 are placed at the same site, namely, a viewingsite B. At the viewing site B, two users, namely, users B1 and B2 areparticipating in the video call.

Each of the videoconference terminal 3 a, the videoconference terminal 3d, the PC 7, and the smartphone 9 is an example of a communicationterminal. OpenGL ES is installed in each of the communication terminals,so that each of the communication terminals can generate predeterminedarea information that represents a partial area of a spherical panoramicimage or generate a predetermined area image from a spherical panoramicimage that is transmitted from any of the other communication terminals.

The arrangement of the terminals (i.e., the communication terminals),the devices (i.e., the displays and the image capturing devices), andthe users illustrated in FIG. 9 is merely an example, and any otherexamples may be employed. For example, an image capturing device capableof capturing a spherical panoramic image may be used instead of theimage capturing device 8 at the viewing site C. In addition, examples ofthe communication terminals include a digital television, a smartwatch,and a car navigation device. In the following description, any arbitraryone of the image capturing devices 1 a and 1 b is referred to as an“image capturing device 1”. In addition, any arbitrary one of thevideoconference terminals 3 a and 3 d is referred to as a“videoconference terminal 3”. Further, any arbitrary one of the displays4 a and 4 d is referred to as a “display 4”.

In the case where the image capturing device 1 can be connected directlyto the communication network 100 by wireless communication or the likeand can distribute data (information) to other devices, the imagecommunication system may be configured so as not to includecommunication terminals such as the videoconference terminals 3 and thesmartphone 9.

Hardware Configurations of Embodiment

Hardware configurations of the image capturing device 1, thevideoconference terminal 3, the communication management system 5, thePC 7, and the smartphone 9 according to the present embodiment aredescribed in detail next with reference to FIGS. 10 to 13. Since theimage capturing device 8 is a general camera, detailed description ofthe image capturing device 8 is omitted herein.

Hardware Configuration of Image Capturing Device

A hardware configuration of the image capturing device 1 is describedfirst with reference to FIG. 10. FIG. 10 is a block diagram illustratinga hardware configuration of the image capturing device 1. In thefollowing description, the image capturing device 1 is a spherical(omnidirectional) image capturing device including two imaging elements.However, the image capturing device 1 may include any number of (two ormore) imaging elements. In addition, the image capturing device 1 is notnecessarily an image capturing device dedicated to omnidirectional imagecapturing, but may be a general digital camera or smartphone to which anexternal omnidirectional image capturing unit is attachable to implementsubstantially the same function as the image capturing device 1.

As illustrated in FIG. 10, the image capturing device 1 includes animager 101, an image processor 104, an imaging controller 105, amicrophone 108, an audio processor 109, a central processing unit (CPU)111, a read-only memory (ROM) 112, a static random access memory (SRAM)113, a dynamic random access memory (DRAM) 114, the operation device115, a network interface (I/F) 116, a communication device 117, anantenna 117 a, and an electronic compass 118.

The imager 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102 b, each having an angle of view of 180 degrees or wider so asto form a hemispherical image.

The imager 101 further includes the two imaging elements 103 a and 103 bcorresponding to the wide-angle lenses 102 a and 102 b, respectively.Each of the imaging elements 103 a and 103 b includes an image sensorsuch as a CMOS sensor or a charge coupled device (CCD) sensor, a timinggeneration circuit, and a group of registers. The image sensor convertsan optical image formed by the fisheye lens 102 a or 102 b into anelectric signal to output image data. The timing generation circuitgenerates horizontal or vertical synchronization signals, pixel clocks,and the like for this image sensor. Various commands, parameters, andthe like for operations of the corresponding imaging element are set inthe group of registers.

Each of the imaging elements 103 a and 103 b of the imager 101 iscoupled to the image processor 104 through a parallel I/F bus. Inaddition, each of the imaging elements 103 a and 103 b of the imager 101is coupled to the imaging controller 105 through a serial I/F bus suchas an inter-integrated circuit (I2C) bus. Each of the image processor104 and the imaging controller 105 is coupled to the CPU 111 through abus 110. The ROM 112, the SRAM 113, the DRAM 114, the operation device115, the network I/F 116, the communication device 117, and theelectronic compass 118 are also coupled to the bus 110.

The image processor 104 obtains pieces of image data output from therespective imaging elements 103 a and 103 b through the parallel I/F busand performs predetermined processing on the pieces of image data. Theimage processor 104 then combines the processed pieces of image data togenerate data representing a Mercator image as illustrated in FIG. 3C.

The imaging controller 105 usually functions as a master device whilethe imaging elements 103 a and 103 b each usually functions as a slavedevice. The imaging controller 105 sets commands and the like in thegroup of registers of the imaging elements 103 a and 103 b through theI2C bus. The imaging controller 105 receives various commands from theCPU 111. The imaging controller 105 obtains status data and the like ofthe group of registers of the imaging elements 103 a and 103 b throughthe I2C bus. The imaging controller 105 then sends the obtained statusdata and the like to the CPU 111.

The imaging controller 105 instructs the imaging elements 103 a and 103b to output the image data at a timing when a shutter button of theoperation device 115 is pressed. The image capturing device 1 can have apreview display function and a movie display function by using a display(for example, a display of the videoconference terminal 3 a). In thiscase, image data are continuously output from the imaging elements 103 aand 103 b at a predetermined frame rate (frames/minute).

As described later, the imaging controller 105 operates in cooperationwith the CPU 111 to function as a synchronization control means thatsynchronize a timing when the imaging element 103 a outputs image dataand a timing when the imaging element 103 b outputs the image data. Inthe present embodiment, the image capturing device 1 does not include adisplay. However, in some embodiments, the image capturing device 1 mayinclude a display.

The microphone 108 converts sound into audio data (signal). The audioprocessor 109 acquires the audio data output from the microphone 108through an I/F bus and performs predetermined processing on the audiodata.

The CPU 111 controls operations of the entire image capturing device 1and performs processing. The ROM 112 stores various programs to beexecuted by the CPU 111. Each of the SRAM 113 and the DRAM 114 is a workmemory, and store programs currently executed by the CPU 111 or datacurrently processed. In particular, in one example, the DRAM 114 storesimage data currently processed by the image processor 104 and processeddata of the Mercator image.

The operation device 115 collectively refers to various operation keys,a power switch, a shutter button, and a touch panel having display andoperation functions. The user operates the operation keys to inputvarious image capturing modes or image capturing conditions.

The network I/F 116 collectively refers to an interface circuit such asa USB I/F with an external medium such as a Secure Digital (SD) card oran external personal computer. The network I/F 116 may be a wired I/F ora wireless I/F. The data of the Mercator image, which is stored in theDRAM 114, is stored in the external medium via the network I/F 116 ortransmitted to an external device such as the videoconference terminal 3a via the network I/F 116 at any desired time.

The communication device 117 communicates with an external device suchas the videoconference terminal 3 a via the antenna 117 a of the imagecapturing device 1 by using a short-range wireless communicationtechnology such as NFC, Bluetooth, or Wi-Fi. The communication device117 is capable of transmitting the data of the Mercator image to anexternal device such as the videoconference terminal 3 a.

The electronic compass 118 calculates an orientation and a tilt (rollangle) of the image capturing device 1 based on the Earth magnetism andoutputs orientation and tilt information. The orientation and tiltinformation is an example of related information (metadata) incompliance with the exchangeable image file format (Exif). Theorientation and tilt information is used in image processing, such asimage correction, of a captured image. The related information alsoincludes data of the image capturing date and time and data size ofimage data, for example.

Hardware Configuration of Videoconference Terminal

A hardware configuration of the videoconference terminal 3 is describednext with reference to FIG. 11. FIG. 1I is a block diagram illustratinga hardware configuration of the videoconference terminal 3. Asillustrated in FIG. 11, the videoconference terminal 3 includes a CPU301, a ROM 302, a RAM 303, a flash memory 304, a solid state drive (SSD)305, a medium I/F 307, operation keys 308, a power switch 309, a busline 310, a network I/F 311, a camera 312, an imaging element I/F 313, amicrophone 314, a speaker 315, an audio input/output I/F 316, a displayI/F 317, an external device coupling I/F 318, a short-rangecommunication circuit 319, and an antenna 319 a for the short-rangecommunication circuit 319.

The CPU 301 controls operations of the entire videoconference terminal3. The ROM 302 stores a program such as an Initial Program Loader (IPL)for driving the CPU 301. The RAM 303 is used as a work area for the CPU301. The flash memory 304 stores a communication program and variouskinds of data such as image data and audio data. The SSD 305 controlsreading or writing of various kinds of data from or to the flash memory304 under control of the CPU 301. Instead of the SSD 305, a hard diskdrive (HDD) may be used. The medium I/F 307 reads or writes (stores)data from or to a recording medium 306 such as a flash memory. Theoperation keys 308 are operated when a destination of a communicationfrom the videoconference terminal 3 is selected. The power switch 309 isa switch that powers on and off the videoconference terminal 3.

The network I/F 311 is an interface that enables communication of datavia the communication network 100 such as the Internet. The camera 312is an example of a built-in image capturing device capable of capturingan image of a subject or object under control of the CPU 301 to obtainimage data. The imaging element I/F 313 is a circuit that controlsdriving of the camera 312. The microphone 314 is an example of abuilt-in sound collecting device that receives sound. The audioinput/output I/F 316 is a circuit for inputting and outputting an audiosignal from and to the microphone 314 and the speaker 315 under controlof the CPU 301. The display I/F 317 is a circuit for transmitting imagedata to the external display 4 under control of the CPU 301. Theexternal device coupling I/F 318 is an interface that couples variousexternal devices to the videoconference terminal 3. The short-rangecommunication circuit 319 is a short-range wireless communicationcircuit in compliance with NFC, Bluetooth, Wi-Fi, or the like.

The bus line 310 includes an address bus and a data bus for electricallycoupling to the components such as the CPU 301 illustrated in FIG. 11 toone another.

The display 4 is an example of a display device, such as a liquidcrystal display or an organic electroluminescence (EL) display thatdisplays an image of an object or subject, an operation icon, and thelike. The display 4 is coupled to the display I/F 317 by a cable 4 c.The cable 4 c may be an analog red green blue (RGB) (video graphic array(VGA)) signal cable, a component video cable, a high-definitionmultimedia interface (HDMI) (registered trademark) signal cable, or adigital video interactive (DVI) signal cable.

The camera 312 includes a lens and a solid-state imaging element thatconverts light into electric charge to convert an image (video) of anobject or subject into electronic data. Examples of the solid-stateimaging element to be used include a CMOS sensor and a CCD sensor. Theexternal device coupling I/F 318 is capable of coupling an externaldevice such as an external camera, an external microphone, or anexternal speaker to the videoconference terminal 3 by a USB cable, forexample. When an external camera is coupled, the external camera isdriven in preference to the built-in camera 312 under control of the CPU301. Likewise, when an external microphone is coupled, the externalmicrophone is driven in preference to the built-in microphone 314 undercontrol of the CPU 301. When an external speaker is coupled, theexternal speaker is driven in preference to the built-in speaker 315under control of the CPU 301.

The recording medium 306 is removable from the videoconference terminal3. The flash memory 304 is replaceable with an electrically erasable andprogrammable ROM (EEPROM), as long as the memory is a nonvolatile memorythat reads or writes data under control of CPU 301.

Hardware Configurations of Communication Management System and PC

Hardware configurations of the communication management system 5 and thePC 7 are described next with reference to FIG. 12. FIG. 12 is a blockdiagram illustrating hardware configurations of the communicationmanagement system 5 and the PC 7. In the present embodiment, both thecommunication management system 5 and the PC 7 are computers and havethe same hardware configuration. Thus, the configuration of thecommunication management system 5 is described below, and thedescription of the configuration of the PC 7 is omitted.

The communication management system 5 includes a CPU 501, a ROM 502, aRAM 503, a hard disk (HD) 504, a hard disc drive (HDD) 505, a mediumdrive 507, a display 508, a network I/F 509, a keyboard 511, a mouse512, a compact disc-rewritable (CD-RW) drive 514, and a bus line 510.The CPU 501 controls operations of the entire communication managementsystem 5. The ROM 502 stores a program such as an IPL used for drivingthe CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD504 stores various kinds of data such as programs for the communicationmanagement system 5. The HDD 505 controls reading or writing of variouskinds of data from or to the HD 504 under control of the CPU 501. Themedium drive 507 controls reading or writing (storing) of data from orto a recording medium 506 such as a flash memory. The display 508displays various kinds of information such as a cursor, a menu, awindow, characters, and an image. The network I/F 509 enablescommunication of data via the communication network 100. The keyboard511 includes a plurality of keys to allow a user to input characters,numbers, and various instructions. The mouse 512 allows a user to selector execute various instructions, select a processing target, and movethe cursor. The CD-RW drive 514 controls reading of various kinds ofdata from a CD-RW 513, which is an example of a removable recordingmedium. The bus line 510 includes an address bus or a data bus andelectrically couples the components described above to one other asillustrated in FIG. 12.

Hardware Configuration of Smartphone

A hardware configuration of the smartphone 9 is described next withreference to FIG. 13. FIG. 13 is a block diagram illustrating a hardwareconfiguration of the smartphone 9. As illustrated in FIG. 13, thesmartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, aCMOS sensor 905, an acceleration and orientation sensor 906, a mediumI/F 908, and a global positioning system (GPS) receiver 909.

The CPU 901 controls operations of the entire smartphone 9. The ROM 902stores a program, such as an IPL, used for driving the CPU 901. The RAM903 is used as a work area for the CPU 901. The EEPROM 904 reads orwrites various kinds of data such as a program for the smartphone 9under control of the CPU 901. The CMOS sensor 905 captures an image of asubject or object (image of interest) under control of the CPU 901 toobtain image data. The acceleration and orientation sensor 906 includesvarious sensors such as an electromagnetic compass for detectinggeomagnetism, a gyrocompass, and an acceleration sensor. The medium I/F908 reads or writes (stores) data from or to a recording medium 907 suchas a flash memory. The GPS receiver 909 receives a GPS signal from a GPSsatellite.

The smartphone 9 also includes a long-range communication circuit 911, acamera 912, an imaging element I/F 913, a microphone 914, a speaker 915,an audio input/output I/F 916, a display 917, an external devicecoupling I/F 918, a short-range communication circuit 919, an antenna919 a for the short-range communication circuit 919, and a touch panel921.

The long-range communication circuit 911 is a circuit that enables thesmartphone 9 to communicate with another device via the communicationnetwork 100. The camera 912 is an example of a built-in image capturingdevice capable of capturing an image of a subject or object undercontrol of the CPU 901 to obtain image data. The imaging element I/F 913is a circuit that controls driving of the camera 912. The microphone 914is an example of a built-in sound collecting device that receives sound.The audio input/output I/F 916 is a circuit for inputting and outputtingan audio signal from and to the microphone 914 and the speaker 915 undercontrol of the CPU 901. The display 917 is an example of a displaydevice, such as a liquid crystal display or an organic EL display thatdisplays an image of an object or subject, various icons, and the like.The external device coupling I/F 918 is an interface for couplingvarious external devices to the smartphone 9. The short-rangecommunication circuit 919 is a short-range wireless communicationcircuit in compliance with NFC, Bluetooth, Wi-Fi, or the like. The touchpanel 921 is an example of an input device for operating the smartphone9 by a user touching the display 917.

The smartphone 9 further includes a bus line 910. The bus line 910includes an address bus and a data bus for electrically coupling thecomponents such as the CPU 901 to one another.

In addition, a recording medium such as a CD-ROM storing each of theabove-described programs or an HD storing these programs can be provideddomestically or overseas as a program product.

Each of the above-described programs may be recorded in a file in aninstallable or executable format on a computer-readable recording mediumor may be downloaded via a network for distribution. Examples of therecording medium include, but not limited to, a compact disc-recordable(CD-R), a digital versatile disc (DVD), a Blu-ray disc (registeredtrademark), an SD card, and a USB memory. Such a recording medium can beprovided domestically or overseas as a program product. For example, thecommunication management system 5 executes a program according to anembodiment of the present disclosure to implement a communicationmanagement method according to an embodiment of the present disclosure.

Functional Configurations of Embodiment

Functional configurations according to the present embodiment aredescribed next with reference to FIGS. 10 to 16I. FIG. 14 and FIG. 15are block diagrams each illustrating functions of part of the imagecommunication system.

Functional Configuration of Image Capturing Device 1 a

As illustrated in FIG. 14, the image capturing device 1 a includes anaccepting unit 12 a, an imaging unit 13 a, a sound collecting unit 14 a,a communication unit 18 a, and a storing and reading unit 19 a. Each ofthese units is a function or means that is implemented as a result ofany of the components illustrated in FIG. 10 operating based on aninstruction from the CPU 111 according to a program for the imagecapturing device, which is loaded from the SRAM 113 to the DRAM 114.

The image capturing device 1 a further includes a storage unit 1000 a,which is implemented by at least one of the ROM 112, the SRAM 113, andthe DRAM 114 illustrated in FIG. 10. The storage unit 1000 a storestherein a globally unique identifier (GUID) of the image capturingdevice 1 a. The storage unit 1000 a includes an image type managementdatabase (DB) 1001 a implemented as an image type management tableillustrated in FIG. 16A.

The image capturing device 1 b includes an accepting unit 12 b, animaging unit 13 b, a sound collecting unit 14 b, a communication unit 18b, a storing and reading unit 19 b, and a storage unit 1000 b, whichimplement substantially the same functions as the functions of theaccepting unit 12 a, the imaging unit 13 a, the sound collecting unit 14a, the communication unit 18 a, the storing and reading unit 19 a, andthe storage unit 1000 a of the image capturing device 1 a. Thus,description of these units is omitted.

Image Type Management Table

FIG. 16A is an illustration of the image type management table. Theimage type management table stores and manages an image data identifier(ID), an Internet protocol (IP) address which is an example of anaddress of a transmission source terminal, and a source name inassociation with one another. The image data ID is an example of imagedata identification information for identifying image data used by theimage capturing device 1 a (1 b) in video communication. The same imagedata ID is assigned to image data transmitted from the same transmissionsource terminal. The image data ID enables a transmission destinationterminal (namely, a reception-side communication terminal) to identifythe transmission source terminal of the received image data. The IPaddress of the transmission source terminal is an IP address of acommunication terminal that transmits image data identified by the imagedata ID associated with the IP address. The source name is a name foridentifying an image capturing device that outputs the image dataidentified by the image data ID associated with the source name, and isan example of image type information. The source name is a namegenerated by each communication terminal such as the videoconferenceterminal 3 a in accordance with a predetermined naming rule.

FIG. 16A indicates that four communication terminals having IP addresses“1.2.1.3”, “1.2.2.3”, “1.3.1.3”, and “1.3.2.3” transmit pieces of imagedata identified by the image data IDs “RS001”, “RS002”, “RS003”, and“RS004”, respectively. FIG. 16A also indicates that the image typesrepresented by the source names of the communication terminals are“Video_Omni”, “Video_Omni”, “Video”, and “Video”, which indicate theimage types of “special image”, “special image”, “general image”, and“general image”, respectively. In the present embodiment, the “specialimage” refers to a spherical panoramic image.

Data other than the image data may be managed in association with theimage data ID. Examples of the data other than the image data includeaudio data and presentation material data to be shared on a screen.

The image type management table (image type management DB 1001) is adata table used when the image capturing device 1 is capable ofcommunicating directly with the communication management system 5 viathe communication network 100. For example, this data table need not beused when the image capturing device 1 a and the videoconferenceterminal 3 a are coupled to each other and the videoconference terminal3 a communicates with the communication management system 5 via thecommunication network 100 as in the first embodiment described later.

Functional Units of Image Capturing Device 1 a

Functional units of the image capturing device 1 a is described indetail next with reference to FIGS. 10 and 14. The accepting unit 12 aof the image capturing device 1 a is mainly implemented by the operationdevice 115 illustrated in FIG. 10 and processing of the CPU 111illustrated in FIG. 10. The accepting unit 12 a accepts an operationinput from a user.

The imaging unit 13 a is mainly implemented by the imager 101, the imageprocessor 104, and the imaging controller 105 illustrated in FIG. 10 andprocessing of the CPU 111 illustrated in FIG. 10. The imaging unit 13 acaptures an image of a scenery to obtain captured image data.

The sound collecting unit 14 a is mainly implemented by the microphone108 and the audio processor 109 illustrated in FIG. 10 and processing ofthe CPU 111 illustrated in FIG. 10. The sound collecting unit 14 acollects sound around the image capturing device 1 a.

The communication unit 18 a is mainly implemented by processing of theCPU 111 illustrated in FIG. 10. The communication unit 18 a is capableof communicating with a communication unit 38 a of the videoconferenceterminal 3 a by using a short-range wireless communication technologysuch as NFC. Bluetooth, or Wi-Fi.

The storing and reading unit 19 a is mainly implemented by processing ofthe CPU 111 illustrated in FIG. 10. The storing and reading unit 19 astores various kinds of data (or information) in the storage unit 1000 aor reads various kinds of data (or information) from the storage unit1000 a.

Functional Configuration of Videoconference Terminal 3 a

As illustrated in FIG. 14, the videoconference terminal 3 a includes atransmission and reception unit 31 a, an accepting unit 32 a, an imageand audio processing unit 33 a, a display control unit 34 a, adetermination unit 35 a, a generation unit 36 a, a calculation unit 37a, a communication unit 38 a, and a storing and reading unit 39 a. Eachof these units is a function or a means that is implemented as a resultof any of the components illustrated in FIG. 11 operating based on aninstruction from the CPU 301 according to a program for thevideoconference terminal 3 a, which is loaded from the flash memory 304to the RAM 303.

The videoconference terminal 3 a further includes a storage unit 3000 awhich is implemented by at least one of the ROM 302, the RAM 303, andthe flash memory 304 illustrated in FIG. 11. The storage unit 3000 aincludes an image type management DB 3001 a, an image capturing devicemanagement DB 3002 a, and a predetermined area management DB 3003 a. Theimage type management DB 3001 a is configured as an image typemanagement table illustrated in FIG. 16B. The image capturing devicemanagement DB 3002 a is configured as an image capturing devicemanagement table illustrated in FIG. 16C. The predetermined areamanagement DB 3003 a is configured as a predetermined area managementtable illustrated in FIG. 16D.

The videoconference terminal 3 d includes a transmission and receptionunit 31 d, an accepting unit 32 d, an image and audio processing unit 33d, a display control unit 34 d, a determination unit 35 d, a generationunit 36 d, a calculation unit 37 d, a communication unit 38 d, a storingand reading unit 39 d, and a storage unit 3000 d. Each of theaforementioned units implements substantially the same function as thefunction of a corresponding one of the transmission and reception unit31 a, the accepting unit 32 a, the image and audio processing unit 33 a,the display control unit 34 a, the determination unit 35 a, thegeneration unit 36 a, the calculation unit 37 a, the communication unit38 a, the storing and reading unit 39 a, and the storage unit 3000 a ofthe videoconference terminal 3 a. Thus, the description of these unitsis omitted. The storage unit 3000 d of the videoconference terminal 3 dincludes an image type management DB 3001 d, an image capturing devicemanagement DB 3002 d, and a predetermined area management DB 3003 d.Each of the aforementioned DBs has substantially the same data structureas the data structure of a corresponding one of the image typemanagement DB 3001 a, the image capturing device management DB 3002 a,and the predetermined area management DB 3003 a of the videoconferenceterminal 3 a. Thus, the description of these DBs is omitted.

Image Type Management Table

FIG. 16B is an illustration of the image type management table. Theimage type management table stores and manages an image data ID, an IPaddress which is an example of an address of a transmission sourceterminal, and a source name in association with one another. The imagedata ID is an example of image data identification information foridentifying image data used in video communication. The same image dataID is assigned to image data transmitted from the same transmissionsource terminal. The image data ID enables a transmission destinationterminal (namely, a reception-side communication terminal) to identifythe transmission source terminal of the received image data. The IPaddress of the transmission source terminal is an IP address of acommunication terminal that transmits image data identified by the imagedata ID associated with the IP address. The source name is a name foridentifying an image capturing device that outputs the image dataidentified by the image data ID associated with the source name, and isan example of image type information. The source name is a namegenerated by each communication terminal such as the videoconferenceterminal 3 a in accordance with a predetermined naming rule.

FIG. 16B indicates that four communication terminals having IP addresses“1.2.1.3”, “1.2.2.3”, “1.3.1.3”, and “1.3.2.3” transmit pieces of imagedata identified by the image data IDs “RS001”, “RS002”, “RS003”, and“RS004”, respectively. FIG. 16A also indicates that the image typesrepresented by the source names of the communication terminals are“Video_Omni”, “Video_Omni”, “Video”, and “Video”, which indicate theimage types of “special image”, “special image”, “general image”, and“general image”, respectively. In the present embodiment, the “specialimage” refers to a spherical panoramic image.

Data other than the image data may be managed in association with theimage data ID. Examples of the data other than the image data includeaudio data and presentation material data to be shared on a screen.

Image Capturing Device Management Table

FIG. 16C is an illustration of the image capturing device managementtable. The image capturing device management table stores and manages avendor ID and a product ID of the GUID of an image capturing devicecapable of obtaining two hemispherical images from which a sphericalpanoramic image is generated. As the GUID, for example, a combination ofa vendor ID (VID) and a product ID (PID) used in a USB device can beused. The vendor ID and the product ID are stored in a communicationterminal such as a videoconference terminal before shipment. In anotherexample, these IDs are added and stored in the videoconference terminalafter shipment.

Predetermined Area Management Table

FIG. 16D is an illustration of the predetermined area management table.The predetermined area management table stores and manages an IP addressof a communication terminal that is a transmission source of capturedimage data, an IP address of a communication terminal that is atransmission destination of the captured image data, and predeterminedarea information representing a predetermined area image currentlydisplayed by the communication terminal that is the transmissiondestination of the captured image data, in association with one another.The communication terminal that is the transmission destination of thecaptured image data is identical to the communication terminal that isthe transmission source of the predetermined area information. Thepredetermined area information is a conversion parameter used forconverting a captured image into an image (predetermined area image) ofthe predetermined area T in this captured image, as illustrated in FIGS.6A, 6B, and 7. The IP address is an example of address information.Other examples of the address information include a Media Access Control(MAC) address and a terminal ID for identifying a correspondingcommunication terminal. Herein, the IP address is represented by asimplified IPv4 address. The IP address may be based on IPv6.

For example, the first to third lines of the predetermined areamanagement table in FIG. 16D manages that the videoconference terminal 3a having an IP address of “1.2.1.3” transmits captured image data, viathe communication management system 5, to the videoconference terminal 3d having an IP address of “1.2.2.3”, the PC 7 having an IP address of“1.3.1.3”, and the smartphone 9 having an IP address of “1.3.2.3”.Further, the predetermined area management table manages that thevideoconference terminal 3 d is a communication terminal that is thetransmission source of predetermined area information (r=10, 0=20,φ=30). Likewise, the predetermined area management table manages thatthe PC 7 is a communication terminal that is the transmission source ofpredetermined area information (r=20, 0=30, φ=40). Furthermore, thepredetermined area management table manages that the smartphone 9 is acommunication terminal that is the transmission source of predeterminedarea information (r=30, 0=40, φ=50).

When the transmission and reception unit 31 a newly receivespredetermined area information including the same set of IP addresses asthe already managed set of IP addresses of the communication terminalthat is the transmission source of captured image data and thecommunication terminal that is the transmission destination of thecaptured image data, the storing and reading unit 39 a overwrites thealready managed predetermined area information with the newly receivedpredetermined area information.

Functional Units of Videoconference Terminal 3 a

Each functional unit of the videoconference terminal 3 a (3 d) isdescribed in more detail next with reference to FIGS. 11, 14, and 15.Since each functional unit of the videoconference terminal 3 d issubstantially the same as the corresponding functional unit of thevideoconference terminal 3 a described below, the description thereof isomitted.

The transmission and reception unit 31 a of the videoconference terminal3 a is mainly implemented by the network I/F 311 illustrated in FIG. 1Iand processing of the CPU 301 illustrated in FIG. 11. The transmissionand reception unit 31 a transmits and receives various kinds of data (orinformation) to and from the communication management system 5 via thecommunication network 100.

The accepting unit 32 a is mainly implemented by the operation keys 308illustrated in FIG. 11 and processing of the CPU 301 illustrated in FIG.11. The accepting unit 32 a accepts various selections or inputs from auser. In another example, an input device such as a touch panel is usedin addition to or in place of the operation keys 308.

The image and audio processing unit 33 a is mainly implemented byinstructions from the CPU 301 illustrated in FIG. 11. The image andaudio processing unit 33 a performs image processing on image dataobtained by capturing an image of a subject or object by the camera 312.After voice of a user is converted into an audio signal by themicrophone 314, the image and audio processing unit 33 a performs audioprocessing on audio data based on this audio signal.

Further, the image and audio processing unit 33 a performs imageprocessing on image data received from another communication terminal,based on the image type information, such as a source name, to enablethe display control unit 34 a to cause the display 4 to display animage. Specifically, when the image type information indicates “specialimage”, the image and audio processing unit 33 a generates data of aspherical panoramic image illustrated in FIG. 48, based on image data(for example, data of hemispherical images illustrated in FIGS. 3A and31), and further generates a predetermined area image illustrated inFIG. 6B. The image and audio processing unit 33 a outputs, to thespeaker 315, an audio signal based on audio data received from anothercommunication terminal via the communication management system 5 tocause the speaker 315 to output sound.

The display control unit 34 a is mainly implemented by the display I/F317 illustrated in FIG. 11 and processing of the CPU 301 illustrated inFIG. 11. The display control unit 34 a causes the display 4 to displayvarious images, characters, and so on.

The determination unit 35 a is mainly implemented by processing of theCPU 301 illustrated in FIG. 11. The determination unit 35 a determinesan image type of image data received from the image capturing device 1a, for example.

The generation unit 36 a is mainly implemented by processing of the CPU301 illustrated in FIG. 11. The generation unit 36 a generates a sourcename, which is an example of the image type information, in accordancewith the aforementioned naming rule, based on a determination resultobtained by the determination unit 35 a indicating “general image” or“special image” (spherical panoramic image in the present embodiment).For example, when the determination unit 35 a determines that the imagetype is “general image”, the generation unit 36 a generates a sourcename “Video” that indicates a “general image” type. On the other hand,when the determination unit 35 a determines that the image type is“special image”, the generation unit 36 a generates a source name“Video_Omni” that indicates a “special image” type.

The calculation unit 37 a is mainly implemented by processing of the CPU301 illustrated in FIG. 11. The calculation unit 37 a calculates theposition (position information) and the direction of a predeterminedarea T1 with respect to a predetermined area T2 in a captured image,based on predetermined area information (i2) that indicates thepredetermined area T2 and predetermined area information (i1) receivedfrom another communication terminal by the transmission and receptionunit 31 a. The predetermined area information (i1) indicates thepredetermined area T1 in the captured image. An image in the case wherethe whole captured image is displayed may be referred to as an “entireimage”.

The communication unit 38 a is mainly implemented by the short-rangecommunication circuit 319 and the antenna 318 a illustrated in FIG. 11and processing of the CPU 301 illustrated in FIG. 11. The communicationunit 38 a is capable of communicating with the communication unit 18 aof the image capturing device 1 a by using a short-range wirelesscommunication technology such as NFC. Bluetooth, or Wi-Fi. Theconfiguration in which the communication unit 38 a and the transmissionand reception unit 31 a individually have a communication unit has beendescribed. In another example, the communication unit 38 a and thetransmission and reception unit 31 a may share a single communicationunit.

The storing and reading unit 39 a is mainly implemented by processing ofthe CPU 301 illustrated in FIG. 11. The storing and reading unit 39 astores various kinds of data (or information) in the storage unit 3000 aor reads various kinds of data (or information) from the memory 3000 a.

Functional Configuration of Communication Management System 5

Each functional unit of the communication management system 5 isdescribed in detail next with reference to FIGS. 12 and 15. Thecommunication management system 5 includes a transmission and receptionunit 51, a recording processing unit 53, a determination unit 55, ageneration unit 56, an image conversion processing unit 57, and astoring and reading unit 59. Each of the these units is a function or ameans that is implemented as a result of any of the componentsillustrated in FIG. 12 operating based on an instruction from the CPU501 in accordance with a program for the communication management system5, which is loaded from the HD 504 to the RAM 503.

The communication management system 5 further includes a storage unit5000 which is implemented by at least one of the ROM 502, the RAM 503,and the HD 504 illustrated in FIG. 12. The storage unit 5000 includes asession management DB 5001, an image type management DB 5002, apredetermined area management DB 5003, a predetermined area managementDB 5003P, and a recorded file storage DB 5004. The session management DB5001 is configured as a session management table illustrated in FIG.16E. The image type management DB 5002 is configured as an image typemanagement table illustrated in FIG. 16F. The predetermined areamanagement DB 5003 is configured as a predetermined area managementtable illustrated in FIG. 16G. The predetermined area management DB5003P is configured as a predetermined area management table Pillustrated in the 16H. The recorded file storage DB 5004 is configuredas a recorded file storage table illustrated in FIG. 16I. The imagecommunication system according to the embodiment may include,independently of the communication management system 5, a recordingserver including the recording processing unit 53, the image conversionprocessing unit 57, the predetermined area management DB 5003, thepredetermined area management DB 5003P, the recorded file storage DB5004, and so on.

Session Management Table

FIG. 16E is an illustration of the session management table. The sessionmanagement table stores and manages a session ID and IP addresses ofparticipant communication terminals in association with each other. Thesession ID is an example of session identification information foridentifying a communication session that enables a video call. Thesession ID is generated for each virtual conference room. The session IDis also managed by each communication terminal, such as thevideoconference terminal 3 a, and is used by each communication terminalto select a communication session. The IP addresses of the participantcommunication terminals indicate IP addresses of the communicationterminals participating in a virtual conference room indicated by thesession ID associated with these IP addresses.

Image Type Management Table

FIG. 16F is an illustration of the image type management table. Theimage type management table manages, in addition to the informationitems managed in the image type management table illustrated in FIG.16B, the same session ID as the session ID managed in the sessionmanagement table in association with one another. The image typemanagement table indicates that three communication terminals having IPaddresses “1.2.1.3”, “1.2.2.3”, and “1.3.1.3” are participating in avirtual conference room indicated by the session ID “se101”. Thecommunication management system 5 manages the same image data ID, thesame IP address of the transmission source terminal, and the same imagetype information as those managed by a communication terminal, such asthe videoconference terminal 3 a to transmit the image type information,etc., to a communication terminal that is currently participating in thevideo call and another communication terminal that newly participates inthe video call when the other communication terminal enters the virtualconference room. This can omit transmission and reception of the imagetype information, etc. between the communication terminal that iscurrently participating in the video call and the communication terminalthat is newly participates in the video call.

Predetermined Area Management Table (Live Recording)

The 16G is an illustration of the predetermined area management tableused in the case of live recording. In live recording, the communicationmanagement system 5 performs real-time distribution (live streamingdistribution) and also uses the predetermined area information of aspherical image displayed at individual sites (also referred to asviewing sites) to perform perspective projection conversion and generatea planar image in real time. The communication management system 5individually records resultant video frames to generate a recorded fileor the like of a viewing state at each viewing site. Live recordingrefers to such a scheme. The predetermined area management table used inthe case of live recording has substantially the same data structure asthe predetermined area management table illustrated in FIG. 16D.However, since the transmission and reception unit 51 transmits thelatest predetermined area information to each communication terminal atregular intervals (for example, at intervals of 30 seconds) as describedbelow, all the pieces of predetermined area information received by thetransmission and reception unit 51 are stored without being deleteduntil the predetermined area information is transmitted at the regularintervals. In FIG. 16G, the more recent predetermined area informationis managed in the upper record.

Predetermined Area Management Table P (Post-Conversion Recording)

FIG. 16H is an illustration of the predetermined area management tableused in the case of post-conversion recording. In post-conversionrecording, the original of each video data and a history of thepredetermined area information of each viewing site alone are recordedduring real-time distribution. In response to a recorded fileacquisition request for requesting post-conversion recording from a userafter the end of real-time distribution, the communication managementsystem 5 generates a recorded file in which the viewing state at eachviewing site is reflected. The predetermined area management table usedin the case of post-conversion recording includes the latestpredetermined area information and an item “timestamp” as compared withthe predetermined area management table used in the case of liverecording. The timestamp is also referred to as time-series informationand can be represented by, for example, UNIX time. The reception dateand time (time) of the predetermined area information received by thetransmission and reception unit 51 from each communication terminal isrecorded. In the case of post-conversion recording, a time-serieshistory is recorded in the field “timestamp” even for the same set ofthe image transmission source and the image transmission destination.

Recorded File Storage Table (Live Recording/Post-Conversion Recording)

FIG. 16I is an illustration of the recorded rile storage table. Therecorded file storage table stores and manages, for each IP address ofthe image transmission source, an IP address of the image transmissiondestination, a live/post fag, and a recorded file name in associationwith one another. The live/post flag is a processing flag (information)for distinguishing whether a recorded file indicated by the recordedfile name (described later) is recorded through live recording orthrough post-conversion recording. The recorded file name indicates arecorded file of a spherical video (image) transmitted by one or moreimage capturing devices 1 serving as image transmission sources in theimage communication system and a recorded file of a scene in which thespherical video (image) is viewed at each viewing site where the deviceor communication terminal indicated by the IP address of each imagetransmission destination is disposed.

In the case of live recording, the communication management system 5records an original video (video or image at the time of real-timedistribution, which is hereinafter simply referred to as original video)for each IP address of the image transmission source. The communicationmanagement system 5 performs real-time distribution of a video (image)captured by the image capturing device 1 and also generates a recordedfile to be viewed with a communication terminal (at a viewing site)indicated by the IP address of each image transmission destination.Specifically, the communication management system 5 generates a recordedfile (1201.mp4) from the viewpoint of a user who is at the viewing siteindicated by the IP address (for example, 1.2.2.1) of the imagetransmission destination, for example. The communication managementsystem 5 also generates a recorded file (1202.mp4) from the viewpoint ofanother user who is at the viewing site indicated by the IP address (forexample, 1.2.2.2), for example. Since the communication managementsystem 5 generates a recorded file to be viewed at each viewing sitealong with distribution of image data in this manner, this scheme isreferred to as a live recording scheme. However, even in this case, theentire spherical video (image) transmitted from the image transmissionsource having the IP address (for example, 1.2.1.3) is desirably left.Thus, the communication management system 5 generates an originalrecorded file indicated by a recorded file name “10101.mp4” withoutdesignating the IP address of the image transmission destination (“−” inthe data table). That is, the file having the recorded file name“10101.mp4” is a file in which a spherical video (image) distributed(transmitted) at the time of live distribution by the image capturingdevice 1 disposed at a transmission source viewing site is recorded asit is. Recorded files having recorded file names “10101.mp4”,“1201.mp4”, and “1202.mp4” are recorded files of live distribution.Thus, the recorded files are viewed at the viewing sites at the sametiming as live distribution.

As described above, in the case of the live recording scheme, thecommunication management system 5 generates a recorded file for eachcombination of the IP address of a video transmission source(distribution source) and the IP address of a video transmissiondestination (distribution destination). That is, information indicatinga video transmitted from which transmission source (distribution source)site is viewed from which viewpoint (predetermined area image) at whichtransmission destination (distribution destination) site is stored. Inaddition, video data (for example, 10101.mp4) transmitted from thetransmission source (distribution source) is stored without beingconverted.

On the other hand, in the case of post-conversion recording, thecommunication management system 5 performs post-conversion recording inresponse to a recorded file acquisition request for requestingpost-conversion recording from a user after an elapse of a predeterminedperiod from live distribution. In this case, since a video (movie file)representing the viewing state at each viewing site is generated laterin response to a request from a user after the end of real-timedistribution, a recorded file name representing the original video ismanaged for each IP address of the image transmission source and arecorded file generated later is added. Information in parentheses inthe field “recorded file name” indicates that the post-conversionrecorded file generated at the time of post-conversion recording and thelive recorded file generated at the time of live distribution have thesame content. For example, as for recorded files associated with the IPaddress “1.2.1.3” of the image transmission source and the IP address“1.2.2.1” of the image transmission destination, the live recorded fileis named “1201.mp4” and the post-conversion recorded file is named“1850.mp4”. These files have different file names from the perspectiveof file management, but the contents of the recorded files are the sameand just the generated timings are different. One of the reasons why thelive recorded file name and the post-conversion recorded file name aremanaged to be different in this manner is different file names areuseful, for example, in management of a distribution fee from theviewpoint of business such as movie distribution service. Specifically,a conceivable reason is that it becomes easier for the imagecommunication system to manage, also with the live/post flag, whichrecorded file is used by the user as the post-conversion recorded file.As described above, when generating a post-conversion recorded file forthe same combination of the IP address of the image transmission sourceand the IP address of the image transmission destination, thecommunication management system 5 may set the name of thepost-conversion recorded file and the name of the live recorded filedifferent or the same.

In the present embodiment, the IP address of the image transmissionsource described above is treated as an example of transmission sourceidentification information or image capturing device identificationinformation. Likewise, the IP address of the image transmissiondestination is treated as an example of transmission destinationidentification information or communication terminal identificationinformation. The image capturing device identification information andthe communication terminal identification information described abovemay be device identification information of the image capturing device 1or terminal identification information of a communication terminal (thesmartphone 9) in addition to the IP address. The image capturing deviceidentification information and the communication terminal identificationinformation are not limited to information on the devices, and may be,for example, user identification information of a user who uses theimage capturing device 1 or user identification information of a userwho uses the communication terminal (the smartphone 9).

Functional Units of Communication Management System

Each functional unit of the communication management system 5 isdescribed in detail next with reference to FIGS. 12 and 15. Thetransmission and reception unit 51 of the communication managementsystem 5 is mainly implemented by the network I/F 509 illustrated inFIG. 12 and processing of the CPU 501 illustrated in FIG. 12. Thetransmission and reception unit 51 transmits and receives various kindsof data (or information) to or from the videoconference terminal 3 a or3 d or the PC 7 via the communication network 100.

The recording processing unit 53 is mainly implemented by processing ofthe CPU 501 illustrated in FIG. 12. In either case of live recording orpost-conversion recording, the recording processing unit 53 performs aprocess of recording an original video based on video (image) datareceived from each communication terminal and stores the recorded videoin an individual recorded file. The recording processing unit 53 thenmanages respective items (bibliographic items) in the recorded filestorage DB 5004 (see FIG. 16I). In the embodiment, it is assumed thatvideo (image) data also includes sound included in the video. In thecase of the live recording scheme, the recording processing unit 53records the original video described above and also records aperspective-projection-conversion-processed image (output image of theimage conversion processing unit 57) that reproduces the viewing stateof each user (viewer) at each viewing site. In the present embodiment,the live recording scheme is referred to as a first recording scheme. Onthe other hand, in the case of the post-conversion recording scheme, inresponse to a recorded file acquisition request for requestingpost-conversion recording from a user who uses a communication terminal,the recording processing unit 53 generates a requested recorded file incooperation with the image conversion processing unit 57 on the basis ofa combination of identification information (for example, an IP address)of a transmission source site and identification information (forexample, an IP address) of a transmission destination site that areincluded in the acquisition request. As another function related topost-conversion recording, the recording processing unit 53 can alsohave a function of providing a user interface (UI) to a viewingrequesting person (hereinafter referred to as a viewing requestor) whorequests viewing of a certain video image on a web page. Note that theviewing requestor may be a user who was viewing a live distributed movieat a predetermined viewing site at the time of live distribution or maybe another user (for example, a director or administrator of thedistributed movie).

In the present embodiment, the post-conversion recording scheme isreferred to as a second recording scheme. In the present embodiment, therecording processing unit 53 functions as an example of recording means.

The determination unit 55 is mainly implemented by processing of the CPU501 illustrated in FIG. 12. The determination unit 55 makes variousdeterminations in the communication management system 5.

The generation unit 56 is mainly implemented by processing of the CPU501 illustrated in FIG. 12. The generation unit 56 generates the imagedata ID.

The image conversion processing unit 57 is mainly implemented byprocessing of the CPU 501 illustrated in FIG. 12. In either case of liverecording or post-conversion recording, the image conversion processingunit 57 performs perspective projection conversion processing on thebasis of a spherical video (image) and predetermined area information togenerate a planar image. In the case of live recording, the imageconversion processing unit 57 performs perspective projection conversionprocessing on a spherical video (image) received from each communicationterminal and supplies the result to the recording processing unit 53 tocause the recording processing unit 53 to record the result.Specifically, the image conversion processing unit 57 extracts, from thepredetermined area management DB 5003 (see FIG. 16G), the predeterminedarea information of each viewing site at which a user is viewing a video(image) based on a spherical video (image), performs the perspectiveprojection conversion processing, and supplies the result to therecording processing unit 53 to cause the recording processing unit 53to record the result. On the other hand, in the case of post-conversionrecording, the image conversion processing unit 57 performs perspectiveprojection conversion processing on the basis of each original videomanaged in the recorded file storage DB 5004 (see FIG. 16I) identifiedby a request from a user and a history of the predetermined areainformation for each timestamp extracted from the predetermined areamanagement DB 5003P (see FIG. 16H), and supplies the result to therecording processing unit 53 to cause the recording processing unit 53to record the result. In the present embodiment, the image conversionprocessing unit 57 functions as an example of image converting means.

The recording processing unit 53 and the image conversion processingunit 57 described above may collectively function as a “recordingconversion processing unit”. In such a case, the recording conversionprocessing unit functions as an example of recording converting means.

The storing and reading unit 59 is mainly implemented by the HDD 505illustrated in FIG. 12 and processing of the CPU 501 illustrated in FIG.12. The storing and reading unit 59 stores various kinds of data (orinformation) in the storage unit 5000 and reads various kinds of data(or information) from the storage unit 5000. In the present embodiment,the storing and reading unit 59 functions as an example of storing andreading means, and the storage unit 5000 functions as an example ofstorage means.

The functional units described above include a group of modules forimplementing the recording function in the communication managementsystem 5. However, a recording server or the like may be providedseparately from the communication management system 5.

Functional Configuration of PC A functional configuration of the PC 7 isdescribed in detail next with reference to FIGS. 12 and 15. The PC 7 hassubstantially the same functional units as those of the videoconferenceterminal 3 a. That is, as illustrated in FIG. 15, the PC 7 includes atransmission and reception unit 71, an accepting unit 72, an image andaudio processing unit 73, a display control unit 74, a determinationunit 75, a generation unit 76, a calculation unit 77, a communicationunit 78, and a storing and reading unit 79. Each of these units is afunction or a means that is implemented as a result of any of thecomponents illustrated in FIG. 12 operating based on an instruction fromthe CPU 701 in accordance with a program for the PC 7, which is loadedfrom the HD 504 to the RAM 503.

The PC 7 further includes a storage unit 7000 which is implemented by atleast one of the ROM 702, the RAM 703, and the HD 704 illustrated inFIG. 12. The storage unit 7000 includes an image type management DB7001, an image capturing device management DB 7002, and a predeterminedarea management DB 7003. The image type management DB 7001, the imagecapturing device management DB 7002, and the predetermined areamanagement DB 7003 have substantially the same data structures as theimage type management DB 3001 a, the image capturing device managementDB 3002 a, and the predetermined area management DB 3003 a,respectively. Thus, the description of these DBs is omitted.

Functional Units of PC

Each functional unit of the PC 7 is described in detail next withreference to FIGS. 12 and 15. The transmission and reception unit 71 ofthe PC 7 is mainly implemented by the network I/F 709 illustrated inFIG. 12 and processing of the CPU 701 illustrated in FIG. 12. Thetransmission and reception unit 71 implements substantially the samefunction as that of the transmission and reception unit 31 a.

The accepting unit 72 is mainly implemented by the keyboard 711 and themouse 712 illustrated in FIG. 12 and processing of the CPU 701illustrated in FIG. 12. The accepting unit 72 implements substantiallythe same function as that of the accepting unit 32 a.

The image and audio processing unit 73 is mainly implemented byinstructions from the CPU 701 illustrated in FIG. 12. The image andaudio processing unit 73 implements substantially the same function asthat of the image and audio processing unit 33 a.

The display control unit 74 is mainly implemented by processing of theCPU 701 illustrated in FIG. 12. The display control unit 74 implementssubstantially the same function as that of the display control unit 34a.

The determination unit 75 is mainly implemented by processing of the CPU701 illustrated in FIG. 12. The determination unit 75 implementssubstantially the same function as that of the determination unit 35 a.

The generation unit 76 is mainly implemented by processing of the CPU701 illustrated in FIG. 12. The generation unit 76 implementssubstantially the same function as that of the generation unit 36 a.

The calculation unit 77 is mainly implemented by processing of the CPU701 illustrated in FIG. 12. The calculation unit 77 implementssubstantially the same function as that of the calculation unit 37 a.

The communication unit 78 is mainly implemented by processing of the CPU701 illustrated in FIG. 12. The communication unit 78 implementssubstantially the same function as that of the communication unit 38 a.

The storing and reading unit 79 a is mainly implemented by processing ofthe CPU 701 illustrated in FIG. 12. The storing and reading unit 79 astores various kinds of data (or information) in the storage unit 7000or reads various kinds of data (or information) from the storage unit7000.

Functional Configuration of Smartphone

A functional configuration of the smartphone 9 is described next withreference to FIGS. 13 and 14. The smartphone 9 has substantially thesame functions as the videoconference terminal 3 a. That is, asillustrated in FIG. 14, the smartphone 9 includes a transmission andreception unit 91, an accepting unit 92, an image and audio processingunit 93, a display control unit 94, a determination unit 95, ageneration unit 96, a calculation unit 97, a communication unit 98, anda storing and reading unit 99. Each of these units is a function or ameans that is implemented as a result of any of the componentsillustrated in FIG. 13 operating based on an instruction from the CPU901 in accordance with a program for the smartphone 9, which is loadedfrom the EEPROM 904 to the RAM 903.

The smartphone 9 further includes a storage unit 9000 which isimplemented by at least one of the ROM 902, the RAM 903, and the EEPROM904 illustrated in FIG. 13. The storage unit 9000 includes an image typemanagement DB 9001, an image capturing device management DB 9002, and apredetermined area management DB 9003. The image type management DB9001, the image capturing device management DB 9002, and thepredetermined area management DB 9003 have substantially the same datastructures as the image type management DB 3001 a, the image capturingdevice management DB 3002 a, and the predetermined area management DB3003 a, respectively. Thus, description of these DBs is omitted.

Functional Units of Smartphone

Each functional unit of the smartphone 9 is described in detail nextwith reference to FIGS. 13 and 14. The transmission and reception unit91 of the smartphone 9 is mainly implemented by the long-rangecommunication circuit 911 illustrated in FIG. 13 and processing of theCPU 901 illustrated in the FIG. 13. The transmission and reception unit91 implements substantially the same function as that of thetransmission and reception unit 31 a.

The accepting unit 92 is mainly implemented by the touch panel 921illustrated in FIG. 13 and processing of the CPU 901 illustrated in FIG.13. The accepting unit 92 implements substantially the same function asthat of the accepting unit 32 a.

The image and audio processing unit 93 is mainly implemented byinstructions from the CPU 901 illustrated in FIG. 13. The image andaudio processing unit 93 implements substantially the same function asthat of the image and audio processing unit 33 a.

The display control unit 94 is mainly implemented by processing of theCPU 901 illustrated in FIG. 13. The display control unit 94 implementssubstantially the same function as that of the display control unit 34a.

The determination unit 95 is mainly implemented by processing of the CPU901 illustrated in FIG. 13. The determination unit 95 implementssubstantially the same function as that of the determination unit 35 a.

The generation unit 96 is mainly implemented by processing of the CPU901 illustrated in FIG. 13. The generation unit 96 implementssubstantially the same function as that of the generation unit 36 a.

The calculation unit 97 is mainly implemented by processing of the CPU901 illustrated in FIG. 13. The calculation unit 97 implementssubstantially the same function as that of the calculation unit 37 a.

The communication unit 98 is mainly implemented by processing of the CPU901 illustrated in FIG. 13. The communication unit 98 implementssubstantially the same function as that of the communication unit 38 a.

The storing and reading unit 99 is mainly implemented by processing ofthe CPU 901 illustrated in FIG. 13. The storing and reading unit 99stores various kinds of data (or information) in the storage unit 90M)or reads various kinds of data (or information) from the storage unit9000.

Processes or Operations of Embodiment

Processes or operations according to the present embodiment is describednext with reference to FIGS. 17 to 29.

Process of Participating in Communication Session

A process of participating in a specific communication session isdescribed first with reference to FIGS. 17 and 18. FIG. 17 is a sequencediagram illustrating the process of participating in a specificcommunication session. FIG. 18 is an illustration of a communicationsession (virtual conference room) selection screen.

In response to a user (e.g., user A1) at the viewing site A performingan operation for displaying a communication session (virtual conferenceroom) selection screen on the videoconference terminal 3 a, theaccepting unit 32 a accepts the operation for displaying thecommunication session selection screen and the display control unit 34 acauses the display 4 a to display the communication session selectionscreen as illustrated in FIG. 18 (step S21). In the session selectionscreen, selection buttons b1, b2, and b3 are displayed. The selectionbuttons b1, b2, and b3 respectively indicate virtual conference roomsR1, R2, and R3, each of which is a selection target. Each of theselection buttons b1, b2, and b3 is associated with a correspondingsession ID.

In response to the user A1 selecting a desired selection button (in thisexample, the selection button b1) for the virtual conference room, theaccepting unit 32 a accepts selection of a corresponding communicationsession (step S22). Then, the transmission and reception unit 31 atransmits, to the communication management system 5, a request forparticipating in the virtual conference room (step S23). Theparticipation request includes a session ID identifying thecommunication session, selection of which is accepted in step S22, andthe IP address of the videoconference terminal 3 a which is a requestsender terminal. Consequently, the transmission and reception unit SI ofthe communication management system 5 receives the participationrequest.

Subsequently, the storing and reading unit 99 adds the IP addressreceived in step S23 in the field of the participant terminal IP addressof a record having the same session ID as the session ID received instep S23 in the session management DB 5001 (see FIG. 16E) to performprocessing for participating in the communication session (step S24).The transmission and reception unit SI transmits a response to theparticipation request to the videoconference terminal 3 a (step S25).The response to the participation request includes the session IDreceived in step S23 and a result of the participation processing.Consequently, the transmission and reception unit 31 a of thevideoconference terminal 3 a receives the response to the participationrequest. A case where the participation processing is successful isdescribed below.

Process of Managing Image Type Information

A process of managing image type information is described next withreference to FIG. 19. FIG. 19 is a sequence diagram illustrating theprocess of managing image type information.

In response to a user (e.g., user A1) at the viewing site A couples thecradle 2 a, on which the image capturing device 1 a is mounted, to thevideoconference terminal 3 a by using a USB cable, the storing andreading unit 19 a of the image capturing device 1 a reads the GUID ofthe image capturing device 1 a from the storage unit 1000 a and thecommunication unit 18 a transmits the GUID of the image capturing device1 a to the communication unit 38 a of the videoconference terminal 3 a(step SSI). Consequently, the communication unit 38 a of thevideoconference terminal 3 a receives the GUID of the image capturingdevice 1 a.

Then, the determination unit 35 a of the videoconference terminal 3 adetermines whether a vendor ID and a product ID that are the same as thevendor ID and the product ID included in the GUID received in step S51are managed in the image capturing device management DB 3002 a (see FIG.16C) to determine the image type (step S52). Specifically, if the samevender ID and the same product ID are managed in the image capturingdevice management DB 3002 a, the determination unit 35 a determines thatthe image capturing device 1 a is an image capturing device thatcaptures a special image (spherical panoramic image in the presentembodiment). On the other hand, if the same vender ID and the sameproduct ID are not managed in the image capturing device management DB3002 a, the determination unit 35 a determines that the image capturingdevice 1 a is an image capturing device that captures a general image.

Then, the storing and reading unit 39 a stores, in the image typemanagement DB 3001 a (see FIG. 16B), the IP address of thevideoconference terminal 3 a, which is a transmission source terminal,in association with the image type information, which is a determinationresult obtained in step S52 (step S53). In this state, no image data IDis associated. The image type information includes, for example, asource name which is determined in accordance with a predeterminednaming rule and an image type (“general image” or “special image”).

Then, the transmission and reception unit 31 a transmits a request foradding the image type information to the communication management system5 (step S54). The request for adding the image type information includesthe IP address of the videoconference terminal 3 a that is thetransmission source terminal and the image type information, both ofwhich are stored in step S53. Consequently, the transmission andreception unit 51 of the communication management system 5 receives therequest for adding the image type information.

Then, the storing and reading unit 59 of the communication managementsystem 5 searches the session management DB 5001 (see FIG. 16E) by usingthe IP address of the transmission source terminal received in step S54as a search key, to read the corresponding session ID (step SSS).

Then, the generation unit 56 generates a unique image data ID (stepS56). Then, the storing and reading unit 59 stores, in the image typemanagement DB 5002 (see FIG. 16F), as a new record, the session ID readin step SSS, the image data ID generated in step S56, and the IP addressof the transmission source terminal and the image type informationreceived in step S54, in association with one another (step S57). Then,the transmission and reception unit 51 transmits the image data IDgenerated in step S56 to the videoconference terminal 3 a. Consequently,the transmission and reception unit 31 a of the videoconference terminal3 a receives the image data ID (step S58).

Then, the storing and reading unit 39 a of the videoconference terminal3 a stores, in the image type management DB 3001 a (FIG. 16B), the imagedata ID received in step S58, in association with the IP address of thevideoconference terminal 3 a that is the transmission source terminaland the image type information that are stored in step S53 (step S59).

The transmission and reception unit 51 of the communication managementsystem 5 transmits a notification of addition of the image typeinformation to another communication terminal (i.e., the videoconferenceterminal 3 d in the present embodiment) (step S60). The notification ofaddition of the image type information includes the image data IDgenerated in step S56, and the IP address of the videoconferenceterminal 3 a that is the transmission source terminal and the image typeinformation that are stored in step S53. Consequently, the transmissionand reception unit 31 d of the videoconference terminal 3 d receives thenotification of addition of the image type information. The destinationof the notification transmitted by the transmission and reception unitSI is indicated by the other IP address associated with the same sessionID as the IP address of the videoconference terminal 3 a in the sessionmanagement DB 5001 (see FIG. 16E). That is, the destination of thenotification is the other communication terminal that is in the samevirtual conference room where the videoconference terminal 3 a is in.

Then, the storing and reading unit 99 of the smartphone 9 stores, in theimage type management DB 9001 (see FIG. 16B), as a new record, the imagedata ID, the IP address of the transmission source terminal, and theimage type information that are received in step S60, in associationwith one another (step S61). Likewise, the notification of addition ofthe image type information is also transmitted to other communicationterminals such as the PC 7 and the smartphone 9, and the PC 7 and thesmartphone 9 store the image type information, etc. in the image typemanagement 7001 and the image type management DB 9001, respectively.Through the process described above, the same information can be sharedamong the communication terminals in the image type management DBs 3001a, 3001 d, 7001 and 9001.

Process of Communicating Captured Image Data

A process of communicating captured image data in a video call isdescribed next with reference to FIG. 20A to FIG. 29. FIGS. 20A and 20Bare illustrations of how a video call is performed. FIG. 20A illustratesthe case where the image capturing device 1 a is not used, while FIG.20B illustrates the case where the image capturing device 1 a is used.

When the image capturing device 1 a is not used as illustrated in FIG.20A and the camera 312 (see FIG. 11) built in the videoconferenceterminal 3 a is used, the videoconference terminal 3 a is to be placedin a corner of a table where images of the users A1 to A4 can becaptured, since the camera 312 has an angle of view that is horizontally125 degrees and vertically 70 degrees. Thus, the users A1 to A4 are tolook in the direction of the videoconference terminal 3 a when theyspeak. Since the users A1 to A4 look in the direction of thevideoconference terminal 3 a, the display 4 a is also to be placed nearthe videoconference terminal 3 a. As a result, since the users A2 andA4, who are away from the videoconference terminal 3 a, are away fromthe microphone 314 (see FIG. 11), the users A2 and A4 are to speak in arelatively loud voice. The users A2 and A4 may also find it difficult tosee contents displayed on the display 4 a.

By contrast, when the image capturing device 1 a is used, the imagecapturing device 1 a can obtain two hemispherical images from which aspherical panoramic image is generated. Thus, the videoconferenceterminal 3 a and the display 4 a can be placed relatively at the centerof the table, as illustrated in FIG. 20B. As a result, since the usersA1 to A4 are close to the microphone 314, the users A1 to A4 can speakin a relatively small voice. It becomes easier for the users A1 to A4 tosee contents displayed on the display 4 a. A whiteboard 6 is provided onthe right side at the viewing site A, and thus the users A1 to A4 canwrite text and drawings on the whiteboard 6.

A process of transmitting captured image data and audio data obtained atthe viewing site A illustrated in FIG. 20B to other communicationterminals (the smartphone 9, the PC 7, and the videoconference terminal3 d) via the communication management system 5 is described next withreference to FIG. 21. FIG. 21 is a sequence diagram illustrating theprocess of communicating captured image data and audio data in a videocall.

The communication unit 18 a of the image capturing device 1 a transmitscaptured image data obtained by capturing an image of a subject orobject or a scenery and audio data obtained by collecting sound to thecommunication unit 38 a of the videoconference terminal 3 a (step S101).In this case, since the image capturing device 1 a is a device capableof obtaining two hemispherical images from which a spherical panoramicimage is generated, the captured image data is constituted by data ofthe two hemispherical images as illustrated in FIGS. 3A and 3B.Consequently, the communication unit 38 a of the videoconferenceterminal 3 a receives the captured image data and the audio data.

Then, the transmission and reception unit 31 a of the videoconferenceterminal 3 a transmits, to the communication management system 5, thecaptured image data and the audio data received from the image capturingdevice 1 a (step S102). Information transmitted at this time includes animage data ID for identifying the captured image data that is atransmission target. Consequently, the transmission and reception unit51 of the communication management system 5 receives the captured imagedata and the image data ID.

Then, the transmission and reception unit 51 of the communicationmanagement system 5 transmits the captured image data and the audio datato communication terminals (the smartphone 9, the PC 7, and thevideoconference terminal 3 d) participating in the same video call inwhich the videoconference terminal 3 a is participating (steps S103,S104, and S105). Information transmitted in these transmission processesincludes the image data ID for identifying the captured image data thatis a transmission target. Consequently, each of the transmission andreception unit 91 of the smartphone 9, the transmission and receptionunit 71 of the PC 7, and the transmission and reception unit 31 d of thevideoconference terminal 3 d receives the captured image data, the imagedata ID, and the audio data.

Examples of a screen displayed on the display 4 d at the viewing site Dis described next with reference to FIGS. 22A to 22C. FIGS. 22A to 22Cillustrate examples of a screen displayed on the display 4 d at theviewing site D. FIG. 22A illustrates the case where an image isdisplayed based on the captured image data transmitted from the imagecapturing device 1 a at the viewing site A via the videoconferenceterminal 3 a and captured image data transmitted from the imagecapturing device 1 b at the viewing site B, without generating aspherical panoramic image and a predetermined area image. On the otherhand, FIG. 22B illustrates the case where a spherical panoramic imageand a predetermined area image are generated based on the captured imagedata transmitted from the image capturing device 1 a and the imagecapturing device 1 b. An image of the viewing site A is displayed in aleft-side display area (layout number “1”) of the display 4 d, and animage of the viewing site B is displayed in an upper-right display area(layout number “2”). Further, an image of the viewing site C isdisplayed in a middle-right display area (layout number “3”) of thedisplay 4 d, an image of the viewing site D is displayed in alower-right display area (layout number “4”). The display area with thelayout number “1” is a main display area, and the display areas with thelayout numbers “2”, “3” and “4” are sub display areas. The image in themain display area and the images in the sub display areas can beswitched in each communication terminal. In general, at each viewingsite, an image of a viewing site where a main person of the video callis present is displayed in the main display area.

When images based on the captured image data transmitted from the imagecapturing devices 1 a and 1 b, each of which is capable of capturing aspherical panoramic image, are displayed as they are, the images of theviewing sites A and B are displayed as illustrated in FIG. 22A, i.e., asa combination of a front-side hemispherical image and a back-sidehemispherical image as illustrated in FIGS. 3A and 3B.

By contrast, when the image and audio processing unit 93 generates aspherical panoramic image and a predetermined area image based on thecaptured image data output from the image capturing devices 1 a and 1 b,each of which is capable of obtaining two hemispherical images fromwhich a spherical panoramic image is generated, the predetermined areaimage, which is a planar image, is displayed as illustrated in FIG. 22B.At the viewing site C, the image capturing device 8 that obtains ageneral image captures an image. At the viewing site D, thevideoconference terminal 3 d that obtains a general image captures animage. Thus, general images (planar images in the present embodiment) ofthe viewing sites C and D are displayed in FIGS. 22A and 22B.

Furthermore, a user at each viewing site is able to change thepredetermined area corresponding to the predetermined area image in thesame spherical panoramic image. For example, in response to the user B1operating the touch panel 921, the accepting unit 92 accepts theoperation for moving the predetermined area image. Thus, the displaycontrol unit 94 can shift, rotate, reduce, or enlarge the predeterminedarea image. Consequently, the predetermined area image in which theusers A1 and A2 at the viewing site A are displayed by default asillustrated in FIG. 22B can be changed to the predetermined area imageillustrated in FIG. 22C. More specifically, in FIG. 22C, thepredetermined area image is changed from one including the users A1 andA2 to another one including the whiteboard 6, in the captured image ofthe viewing site A illustrated in FIG. 20B.

Note that each of sphere icons 191 and 192 illustrated in FIGS. 22B and22C is an example of a special image identification icon indicating thatan image being displayed is a predetermined area image representing thepredetermined area T, which is a part of a spherical panoramic image.The display positions of the sphere icons 191 and 192 may be anywheresuch as an upper left corner, lower left corner, or lower right cornerinstead of an upper right corner. In addition, a type of each of thesphere icons 191 and 192 is not limited to the one illustrated in FIGS.22B and 22C. For example, in alternative to or in addition to the sphereicons 191 and 192, text such as “Spherical Image” or a combination ofthe icon and the text may be used.

A process performed by the image communication system when apredetermined area image is displayed as illustrated in FIG. 22B and thepredetermined area image is changed as illustrated in FIG. 22C from theone illustrated in FIG. 22B is described next with reference to FIG. 23.FIG. 23 is a sequence diagram illustrating a process of sharingpredetermined area information. In FIG. 23, the videoconference terminal3 a at the viewing site A is a third communication terminal, thevideoconference terminal 3 d at the viewing site D is anothercommunication terminal, and the smartphone 9 at the viewing site B is acommunication terminal (terminal of interest).

First, when the user D1, D2 or D3 operates the videoconference terminal3 d at the viewing site D to display the predetermined area image of theviewing site A as illustrated in FIG. 22B, the transmission andreception unit 31 d of the videoconference terminal 3 d transmits, tothe communication management system 5, predetermined area informationrepresenting the predetermined area image currently displayed (stepS111). This predetermined area information includes the IP address ofthe videoconference terminal 3 a, which is a transmission sourceterminal of the captured image data, and the IP address of thevideoconference terminal 3 d, which is a transmission destinationterminal of the captured image data and the transmission source of thepredetermined area information. Consequently, the transmission andreception unit 51 of the communication management system 5 receives thepredetermined area information.

The storing and reading unit 59 of the communication management system 5stores, in the predetermined area management DB 5003, the predeterminedarea information, the IP address of the transmission source terminal,and the IP address of the transmission destination terminal which arereceived in step S111, in association with one another (step S112). Theprocessing of step S111 and step S112 is performed each time thepredetermined area image is changed in the videoconference terminal 3 d,for example, from the one illustrated in FIG. 22B to the one illustratedin FIG. 22C.

The storing and reading unit 59 of the communication management system 5reads, from among a plurality of sets of predetermined area informationand IP addresses stored in the predetermined area management DB 5003,the latest (the most recently stored) set of predetermined areainformation and IP addresses at regular intervals (for example, atintervals of 30 seconds) (step S113). Then, the transmission andreception unit 51 distributes (transmits) the predetermined areainformation including the IP addresses read in step S113, to othercommunication terminals (i.e., the videoconference terminal 3 a, thesmartphone 9, and the PC 7) participating in the same video call inwhich the videoconference terminal 3 d, which is the transmission sourceterminal of the predetermined area information, is participating (stepsS114, S116, and S118). Consequently, the transmission and reception unit31 a of the videoconference terminal 3 a receives the predetermined areainformation. Then, the storing and reading unit 39 a stores thepredetermined area information and the IP addresses received in stepS114 in association with one another in the predetermined areamanagement DB 3003 a (step S115). Likewise, in the smartphone 9, afterthe transmission and reception unit 91 receives the predetermined areainformation, the storing and reading unit 99 stores the predeterminedarea information and the IP addresses received in step S116 in thepredetermined area management DB 9003 in association with one another(step S117). Further, in the PC 7, after the transmission and receptionunit 71 receives the predetermined area information, the storing andreading unit 79 stores, in the predetermined area management DB 7003,the predetermined area information received in step S118 in associationwith the IP addresses received at the same time (step S119).

As described above, the predetermined area information representing thepredetermined area image changed at the viewing site A is transmitted toeach of the communication terminals at the other viewing sites B, C, andD participating in the same video call. As a result, the predeterminedarea information representing the predetermined area image currentlydisplayed at the viewing site A is shared among the other communicationterminals at the other viewing sites B, C, and D. This process isperformed in substantially the same manner, when the predetermined areaimage is changed at any of the sites B, C, and D. Accordingly, thepredetermined area information representing the predetermined area imagedisplayed at any one of the sites is shared by the communicationterminals at the other sites that are in the same video call.

A method of using the predetermined area information shared at the sitesis described next with reference to FIGS. 24 to 29. FIG. 24 is aflowchart illustrating a process of displaying a predetermined areaimage. Since the processes performed by the communication terminals arethe same, the process performed by the smartphone 9 at the site B isdescribed here. Specifically, a process performed by the smartphone 9 atthe viewing site B in response to the videoconference terminal 3 dtransmitting predetermined area information representing a predeterminedarea image to other communication terminals in the same video call whenthe videoconference terminal 3 d at the viewing site D displays apredetermined area image based on captured image data transmitted fromthe videoconference terminal 3 a at the site A.

First, the storing and reading unit 99 of the smartphone 9 searches theimage type management DB 9001 (see FIG. 16B) using the image data IDreceived in step S103 in FIG. 21 as a search key, to read thecorresponding source name (image type information) (step S131).

Next, the determination unit 95 determines whether the image typeinformation read in step S131 indicates “special image” (step S132). Ifthe image type information indicates “special image” (YES in step S132),the storing and reading unit 99 further searches the predetermined areamanagement DB 9003 for predetermined area information representing apredetermined area image displayed by each of the communicationterminals at the other sites (step S133). Then, the determination unit95 determines whether the predetermined area information representingthe predetermined area image displayed by each of the communicationterminals at the other sites is managed in the predetermined areamanagement DB 9003 (step S134). If the predetermined area informationrepresenting the predetermined area image displayed by each of thecommunication terminals at the other sites is managed (YES in stepS134), the calculation unit 97 calculates the position of the point ofgaze of the predetermined area T1 with respect to the predetermined areaT2 in the entire image, based on predetermined area information (i2)representing the predetermined area image of the predetermined area T2displayed by the smartphone 9 (terminal of interest) and thepredetermined area information (i1) representing the predetermined areaimage of the predetermined area T1 which is received by the transmissionand reception unit 91 from the other communication terminal and managedin the predetermined area management DB 9003. Likewise, the calculationunit 97 calculates the direction of the predetermined area T1 withrespect to the predetermined area T2 in the entire image (step S135).Strictly speaking, the position in this case indicates the point of gazeof the predetermined area T1 with respect to the point of gaze of thepredetermined area T2. The point of gaze is the center point asdescribed above but may be an upper left end (or a lower left end, anupper right end, or a lower right end) of a rectangle of eachpredetermined area. The point of gaze may be a specific point in eachpredetermined area.

A method of calculating the point of gaze of the predetermined area T1with respect to the predetermined area T2 in the entire image isdescribed with reference to FIGS. 25A and 25B. FIG. 25A is a diagramillustrating definitions of angles of a virtual camera. FIG. 25B is adiagram illustrating a method of calculating the position of the pointof gaze at another site in the predetermined area image at the site ofinterest by using parallel projection viewed from the above.

As illustrated in FIG. 25A, the calculation unit 97 obtains the radiusvector r, the polar angle θ, and the azimuth angle φ from thepredetermined area information representing the predetermined area imagedisplayed by the display control unit 94 of the smartphone 9, and setsthe obtained values as CP1(r0, 01, φ1). Then, the calculation unit 97obtains the radius vector r, the polar angle θ, and the azimuth angle φfrom the predetermined area information of the other site read in stepS133, and sets the obtained values as CP2(r0, 02, φ2).

When the predetermined area T2 centered at the point of gaze CP1 of thesmartphone 9 is considered, a width w and a height h of thepredetermined area T2 are respectively represented by a width w and aheight hcos01 in FIG. 25B that illustrates parallel projection from thepolar direction.

Since the radius vector of the point of gaze CP1 is projected to be alength r0 sin θ1 and the radius vector of the point of gaze CP2 isprojected to be a length r0 sin θ2, the point of gaze CP1 is located atcoordinates (r0 sin θ1·r0 cos φ1, r0 sin θ1·r0 sin φ1) and the point ofgaze CP2 is located at coordinates (r0 sin 02·r0 cos φ2, r0 sin 02·r0cos φ2).

Since the coordinates of the point of gaze CP1 and the coordinates ofthe point of gaze CP2 are successfully derived in FIG. 25B in the mannerdescribed above, the position of the point of gaze CP2 in the plane ofthe predetermined area T2 having the width w and the height h can bederived by using general coordinate conversion.

A method of calculating the direction of the predetermined area T1 withrespect to the predetermined area T2 in the entire image is describedwith reference to FIGS. 26A and 26B. FIG. 26A is a diagram illustratingdefinitions of angles, and FIG. 26B is a diagram illustratingdefinitions of angle ranges.

As illustrated in FIG. 26A, the calculation unit 97 acquires the azimuthangle φ from the predetermined area information representing thepredetermined area image displayed by the display control unit 94 of thesmartphone 9, and sets this azimuth angle φ as a rotation angle φ1.Then, the calculation unit 97 acquires the azimuth angle φ from thepredetermined area information of the other site read out in step S133,and sets this azimuth angle φ as a rotation angle φ2. Further, thecalculation unit 97 sets a difference between the rotation angle φ2 andthe rotation angle φ1 as a rotation angle 93.

Suppose that α1 denotes an angle range having the center at the rotationangle φ of the site of interest and α2 denotes an angle range having thecenter at an angle obtained by adding 180 degrees to a horizontal angleof the site of interest, as illustrated in FIG. 26B. In this case, thecalculation unit 97 calculates the direction of the predetermined areaT1 with respect to the predetermined area T2 in the entire image in thefollowing manner. (1) When the rotation angle φ3 is included in theangle range α1, the positional relationship is determined as “forwarddirection”. (2) When the rotation angle φ3 is included in the anglerange α2, the positional relationship is determined as “backwarddirection”. (3) When the rotation angle φ3 is included in an angle rangewhere a value obtained by subtracting a half value of a fixed angle αfrom a reference horizontal is greater than 0 degree and less than 180degrees, the positional relationship is determined as “rightwarddirection”. (4) When the rotation angle φ3 is included in an angle rangewhere a value obtained by subtracting the fixed angle α from thereference horizontal angle is greater than 0 degree and less than 180degrees, the positional relationship is determined as “rightwarddirection”. (5) When the rotation angle φ3 is included in an angle rangewhere a value obtained by subtracting the fixed angle α from thereference horizontal angle is greater than 180 degree and less than 360degrees, the positional relationship is determined as “leftwarddirection”.

Referring back to FIG. 24, the image and audio processing unit 93generates a predetermined area image including a point-of-gaze markindicating the point of gaze calculated by the calculation unit 97 and adisplay direction mark indicating the direction calculated by thecalculation unit 97 (step S136). The display position of thepoint-of-gaze mark is directly determined from the position of thepredetermined area T1 with respect to the predetermined area T2 in theentire image. The display position of the display direction mark isdetermined through each of the determinations (1) to (5) described aboveby using the position of the predetermined area T1 with respect to thepredetermined area T2 in the entire image.

At this time, since the image type information indicates “specialimage”, the image and audio processing unit 93 combines the sphere icons191 and 192 indicating a spherical panoramic image with the respectivepredetermined area images. Then, as illustrated in FIGS. 27A, 27B, 27C,28A, 28B, and 29, the display control unit 94 displays the predeterminedarea image generated in step S136 (step S137). FIGS. 27A, 27B and 27Cillustrate images displayed in the main display area, which are threedisplay examples of the predetermined area image including the displaydirection marks. FIGS. 28A and 28B illustrate images displayed in themain display area, which are two display examples of the predeterminedarea image including the point-of-gaze marks. FIG. 29 illustrates animage displayed in the main display area, which is one display exampleof the predetermined area image including the point-of-gaze mark and thedisplay direction marks. Images of all the viewing sites are displayedas illustrated in FIGS. 22A, 22B, and 22C during a video call. However,in FIGS. 27A, 27B, 27C, 28A, 28B, and 29, the image of the viewing siteA alone is displayed because of a limitation on the area of the drawing.

As illustrated in FIG. 27A, in the predetermined area image that is partof the image of the viewing site A, display direction marks m11, m13,and m14 indicating directions of the predetermined area images displayedat the other sites with respect to the predetermined area imagedisplayed at the viewing site B (site of interest) in the entire imageare displayed.

Display direction marks m21, m23, and m24 illustrated in FIG. 27Bcorrespond to the display direction marks m11, m13, and m14 illustratedin FIG. 27A, respectively. Further, display direction marks m31, m33 andm34 illustrated in FIG. 27C also correspond to the display directionmarks m11, m13, and m14 illustrated in FIG. 27A, respectively.

Each display direction mark described above is an example of directioninformation. The direction information may be represented in any othersuitable form. In another example, the direction information isindicated by characters such “right”. “left”, “back” and “front” insteadof by an arrow.

As illustrated in FIG. 28A, in the predetermined area image that is apart of the image of the viewing site A, point-of-gaze marks m41 and m42indicating points of gaze of the predetermined area images displayed atthe other sites are displayed with respect to the predetermined areaimage displayed at the viewing site B (site of interest) in the entireimage. The point-of-gaze mark may be displayed semi-transparently so asnot to hide the predetermined area image. Point-of-gaze marks m51 andm52 illustrated in FIG. 28B correspond to the point-of-gaze marks m41and m42 illustrated in FIG. 28A, respectively.

In FIG. 28A, a label “C” indicating the viewing site name is displayedfor the point-of-gaze mark m41 and a label “D” indicating the viewingsite name is displayed for the point-of-gaze mark m42 so that theviewing sites corresponding to the points of gaze can be distinguishedbetween. By contrast, in FIG. 28B, the viewing site names are notdisplayed but different viewing sites are indicated by point-of-gazemarks having different patterns (such as a grid pattern and an obliqueline pattern). In this case, the filling pattern is changed for eachviewing site so that the points of gaze of the other sites can bedistinguished between. If a table that associates a pattern with aviewing site name is prepared, a user at each viewing site can identifythe viewing site from the pattern of the point-of-gaze mark. This tablemay be printed on paper or may be stored as electronic data at eachviewing site.

The point-of-gaze marks may be distinguished between by changing thecolor or line type instead of the displayed pattern. The point-of-gazemark is an example of corresponding position information.

In FIG. 29, the point-of-gaze mark m41 indicating the point of gaze isdisplayed when the point of gaze of the other site is included in therange of the predetermined area image, and the display direction marksm11 and m14 are displayed when the points of gaze of the other sites arenot included in the range of the predetermined area image.

Referring back to FIG. 24, if it is determined in step S134 that thepredetermined area information representing the predetermined area imagedisplayed by the communication terminal at the other site is not managedin the predetermined area management DB 9003 (NO in step S134), theimage and audio processing unit 93 generates the predetermined areaimage without including the point-of-gaze mark and the display directionmark (step S138). Then, the process proceeds to step S137.

On the other hand, if it is determined in step S132 that the image typeinformation does not indicate “special image” (NO in step S132), i.e.,if the image type information indicates “general image”, the image andaudio processing unit 93 does not generate a spherical panoramic imagefrom the captured image data received in step S103 and the displaycontrol unit 94 displays a general image (step S139).

As described above, the users B1 and B2 at the viewing site B canrecognize a positional relationship between the predetermined area imagedisplayed at the viewing site B and the predetermined area imagesdisplayed at the other sites. This may prevent the users B1 and B2 atthe viewing site B from being unable to keep up with discussion in aconference or the like.

As described above, the communication terminal, such as thevideoconference terminal 3 a, according to the present embodimentgenerates a spherical panoramic image and a predetermined area imagebased on an image data ID transmitted with image data and based on thecorresponding image type information. This may prevent the front-sidehemispherical image and the back-side hemispherical image from beingdisplayed as illustrated in FIG. 22A.

Further, a user at any viewing site can recognize which part of theentire spherical panoramic image is displayed as the predetermined areaimage at the other site. This makes it easier for the users to keep upwith discussion in a conference or the like as compared with the relatedart.

Further, in FIG. 23, if the communication management system 5 transferspredetermined area information received from the videoconferenceterminal 3 d to the other communication terminals each time thecommunication management system 5 receives the predetermined areainformation from the videoconference terminal 3 d, the users B1 and B2may be unable to concentrate on the video call because of flickering ofthe point-of-gaze mark and the display direction marks illustrated inFIG. 29. Accordingly, as in steps S112 to S114 described above, thecommunication management system 5 distributes, at regular intervals, thelatest set of predetermined area image and IP addresses at that timepoint. This allows each user to concentrate on the video call.

Second Embodiment

A second embodiment is described next with reference to FIG. 30. FIG. 30is a sequence diagram illustrating another example of the processillustrated in FIG. 23, i.e., another process of sharing predeterminedarea information. In FIG. 30, the videoconference terminal 3 a at theviewing site A is a communication terminal (terminal of interest), andthe videoconference terminal 3 d at the viewing site D is anothercommunication terminal.

In the first embodiment described above, as illustrated in FIG. 23, thecommunication management system 5 temporarily stores predetermined areainformation transmitted from each of the communication terminals (seestep S112) and transmits the predetermined area information at regularintervals to each of the communication terminals other than thetransmission source (see steps S114 to S119). By contrast, in thepresent embodiment, as illustrated in FIG. 30, instead of thecommunication management system 5, the transmission source communicationterminal (the videoconference terminal 3 a in the present embodiment) ofcaptured image data temporarily stores the predetermined areainformation (see step S213), and transmits the predetermined areainformation to each of the communication terminals other than theterminal of interest (i.e., the videoconference terminal 3 a) at regularintervals (see steps S215 to S221). That is, in the present embodiment,a communication terminal that is a transmission source of captured imagedata manages how a predetermined area image representing thepredetermined area T1 is displayed by another communication terminalbased on the captured image data transmitted from the terminal ofinterest (i.e., the videoconference terminal 3 a in the presentembodiment).

The configuration of the present embodiment is the same as that of thefirst embodiment except for the process illustrated in FIG. 23.Therefore, in the following, the same reference signs are used for thesame configurations to omit description. A different processing portionfrom the first embodiment is described with reference to FIG. 30.

First, when the user D1 operates the videoconference terminal 3 d at theviewing site D to display a predetermined area image of the viewing siteA, the transmission and reception unit 31 d of the videoconferenceterminal 3 d transmits, to the communication management system 5,predetermined area information representing the displayed predeterminedarea image (step S211). This predetermined area information includes theIP address of the videoconference terminal 3 a, which is a transmissionsource terminal of the captured image data, and the IP address of thevideoconference terminal 3 d, which is a transmission destinationterminal of the captured image data and the transmission source of thepredetermined area information. Consequently, the transmission andreception unit 51 of the communication management system 5 receives thepredetermined area information.

Next, the transmission and reception unit 51 of the communicationmanagement system 5 transmits the predetermined area informationincluding the IP addresses received in step S211, to the videoconferenceterminal 3 a which is a transmission source terminal of the capturedimage data (step S212). Consequently, the transmission and receptionunit 31 a of the videoconference terminal 3 a receives the predeterminedarea information.

Then, the storing and reading unit 39 a of the videoconference terminal3 a stores, in the predetermined area management DB 3003 a, thepredetermined area information, the IP address of the transmissionsource terminal, and the IP address of the transmission destinationterminal which are received in step S212, in association with oneanother (step S213). This processing of step S213 is processing ofmanaging how the captured image data transmitted from the terminal ofinterest (the videoconference terminal 3 a in this case) is displayed inthe other communication terminals. The processing of steps S211 and S213is performed each time the predetermined area image is changed in thevideoconference terminal 3 d.

The storing and reading unit 39 a of the videoconference terminal 3 areads, from among a plurality of sets of predetermined area informationand IP addresses stored in the predetermined area management DB 3003 a,the latest (the most recently stored) set of predetermined areainformation and IP addresses at that time point, at regular intervalssuch as at intervals of 30 seconds (step S214). Then, the transmissionand reception unit 31 a transmits the predetermined area informationincluding the IP addresses read in step S214, to the communicationmanagement system 5 (step S215). Consequently, the transmission andreception unit 51 of the communication management system 5 receives thepredetermined area information including the IP addresses.

Then, the transmission and reception unit 51 of the communicationmanagement system 5 transmits (distributes) the predetermined areainformation including the IP addresses received in step S215, to each ofthe communication terminals (i.e., the videoconference terminal 3 d, thesmartphone 9, and the PC 7) (steps S216, S218, and S220). Consequently,the transmission and reception unit 31 d of the videoconference terminal3 d receives the predetermined area information. The storing and readingunit 39 d stores, in the predetermined area management DB 3003 d, thepredetermined area information received in step S216 in association withthe IP addresses received at the same time (step S217). Likewise, thetransmission and reception unit 91 of the smartphone 9 receives thepredetermined area information. Then, the storing and reading unit 99stores, in the predetermined area management DB 9003, the predeterminedarea information received in step S218 in association with the IPaddresses received at the same time (step S219). Further, thetransmission and reception unit 71 of the PC 7 receives thepredetermined area information. Then, the storing and reading unit 79stores, in the predetermined area management DB 7003, the predeterminedarea information received in step S220 in association with the IPaddresses received at the same time (step S221).

As described above, according to the present embodiment, a communicationterminal (terminal of interest) that is a transmission source ofcaptured image data collects predetermined area information indicatinghow each communication terminal displays an image based on the capturedimage data transmitted from the terminal of interest, and distributesthe collected predetermined area information to each communicationterminal. Consequently, in addition to the effects of the firstembodiment, concentration of a load on the communication managementsystem 5 can be avoided in the case where a large number ofcommunication terminals are participating in the same videoconference orthe like.

Third Embodiment

A third embodiment is described next.

In FIG. 10, the image capturing device 1 a includes a single microphone108. However, in the present embodiment, the image capturing device 1 aincludes a plurality of directional microphones. When the imagecapturing device 1 a uses the plurality of directional microphones,audio data of each directional microphone is transmitted from the imagecapturing device 1 a to the videoconference terminal 3 a. Consequently,the calculation unit 37 a of the videoconference terminal 3 a calculatesthe direction of a sound source (microphone angle) based on the audiodata of each microphone. The calculated direction is used to identifythe position of a speaker (at a transmission source of the capturedimage). The same applies to the image capturing device 1 b. The imagecapturing device 1 b including a plurality of directional microphonestransmits audio data of each microphone to the smartphone 9. Thecalculation unit 97 of the smartphone 9 calculates a direction of asound source (microphone angle) based on the audio data of eachmicrophone. The calculated angle is used to identify the position of aspeaker (at a transmission source of the captured image).

As described above, according to the present embodiment, a predeterminedarea image of which portion of the entire spherical panoramic image isdisplayed at the other site can be recognized and thus the predeterminedarea image can be displayed more accurately.

Fourth Embodiment

A fourth embodiment is described next with reference to FIGS. 31 to 35.FIG. 31 is a sequence diagram illustrating a process of communicatingcaptured image data and audio data in a video call. The presentembodiment differs from the first embodiment in that processingperformed by the videoconference terminal 3 a can be omitted from theprocessing illustrated in FIG. 21. It is assumed that the predeterminedarea management DB 5003 (see FIG. 16G) used in the first embodiment tothe third embodiment is given predetermined values through each processillustrated in FIG. 23 or the like also in the present embodiment. Asdescribed above, in the present embodiment, description of substantiallythe same configurations as those of the other embodiments is omitted andprocesses newly illustrated in FIG. 31 and subsequent figures aredescribed. On the assumption that hardware resources and functionalunits constituting the image communication system described above areused in the fourth embodiment, the case where an event held outdoorssuch as a construction site or the like is live-distributed and alive-distributed video (image) is viewed using one or more communicationterminals disposed at remote sites.

It is assumed that the image capturing device 1 a is an image capturingdevice that transmits captured image data representing a captured imageto the communication management system 5 and that is capable ofperforming real-time distribution (live streaming distribution) of thecaptured image to other communication terminals, at the viewing site Aillustrated in FIG. 9. That is, in the first embodiment, captured imagedata representing a spherical panoramic image captured with the imagecapturing device 1 a is transmitted (distributed) via thevideoconference terminal 3 a (communication terminal). Instead, in thefourth embodiment, the image capturing device 1 a that captures aspherical image is capable of transmitting (distribute through livestreaming) captured image data representing the captured sphericalpanoramic image to other communication terminals provided at other sites(viewing site B, C, D, . . . ) including the viewing site A withoutprocessing the captured image data.

In FIG. 31, first, the communication unit 18 a of the image capturingdevice 1 a transmits captured image (video) data obtained by capturingan image of a subject or object, scenery, or the like and audio dataobtained by collecting sound, to the transmission and reception unit SIof the communication management system 5 (step S301). Consequently, thetransmission and reception unit 51 of the communication managementsystem 5 receives the captured image data and the audio data ofcollected sound that are transmitted by the image capturing device 1 a.At this time, the captured image (video) data includes the image dataID. Since the image capturing device 1 a is capable of obtaining twohemispherical images from which a spherical panoramic image is obtained,the captured image (video) data is constituted by data of the twohemispherical images as illustrated in FIGS. 3A and 3B. In the presentembodiment, for convenience of description, it is assumed that thecaptured image (video) data includes the audio data described above andthat the term “captured image (video) data” indicates that the audiodata is included. The captured image (video) data may also be simplyreferred to as captured image data.

Then, the communication management system 5 performs a recording processon the received captured image data (step S302). Note that contents ofstep S302 is described in detail in a flow of the recording processbelow.

Flow of Recording Process

Flow of Recording Process during Live Recording A process of liverecording performed by the communication management system 5 isdescribed next. FIG. 32 is a flowchart illustrating a process ofselecting live recording or post-conversion recording and of performinglive recording. The process illustrated in FIG. 32 is performed eachtime media data is received from each communication terminal. Thedetermination unit 55 first determines whether the received data isvideo data (step S302-1).

If the determination unit 55 determines that the received data is notvideo data (NO in step S302-1), the process exits from this flow. On theother hand, if the determination unit 55 determines that the receiveddata is video data (YES in step S302-1), the recording processing unit53 operates in cooperation with the storing and reading unit 59 to storevideo frames of the received video data as an original recorded file inthe recorded fic storage DB 5004 (sec FIG. 16I) (step S302-2). In thisprocessing step, the recorded file is recorded in a video format (forexample, the equirectangular format) including the entire sphere even inthe case of a spherical panoramic video.

Then, the determination unit 55 determines whether the received videoframes have been recorded through live recording or post-conversionrecording (step S302-3). If the received video frames are recordedthrough post-conversion recording (POST in step S302-3), thedetermination unit 55 exits from this flow to transition to a processindicated by a circled A in FIG. 33 as a subsequent process.

On the other hand, if the received video frames are recorded throughlive recording (LIVE in step S302-3), the determination unit 55 furthersearches the image type management DB 5002 (see FIG. 16F) by using theimage data ID included in the received data as a search key to read thecorresponding image type and determines whether the video based on thereceived data is a spherical video (step S302-4). If the video based onthe received data is not a spherical video but is a planar video (NO instep S302-4), the determination unit 55 dues not perform any processingand the process exits from this flow.

On the other hand, if the video based on the received data is aspherical video (YES in step S302-4), the storing and reading unit 59searches the predetermined area management DB 5003 (see FIG. 16G) byusing each IP address of the image transmission destination indicatingthe corresponding viewing site as the search key to read the latestpredetermined area information corresponding to each IP address relatedto the received data (step S302-5).

Then, the image conversion processing unit 57 performs perspectiveprojection conversion for each piece of the read latest predeterminedarea information to generate a predetermined area image (planar image)(step S302-6).

After the predetermined area images are generated, the storing andreading unit 59 stores the generated predetermined area images (planarimages) as video frames in a recorded file in the recorded file storageDB 5004 (see FIG. 16I) (step S302-7). The process then exits from thisflow.

The processing of step S302-5 to S302-07 described above is performedfor data of each viewing site where the video based on the receivedvideo data is viewed, and a video file that reflects the viewing stateand is given a recorded file name is generated and managed for eachviewing site. In this manner, in the case of the live recording scheme,the communication management system 5 is capable of generating arecorded file that reflects the viewing state of each of all the viewingclients (viewing sites) as well as performing real-time distribution ofthe captured images. At that time, screen capturing or the like is notlonger to be performed at each viewing client (each communicationterminal), and a processing load on the viewing client and generation ofdata from the viewing client to the communication management system 5can be suppressed. The captured original video is also stored with apredetermined recorded file name in the recorded file storage DB 5004(see FIG. 16I). Therefore, a record is left for an area to which noattention is paid at any viewing site in the spherical video. Thisenables the user to refer to the left record after an elapse of apredetermined period.

Flow of Recording Process during Post-Conversion Recording Apost-conversion recording process performed by the communicationmanagement system 5 is described next. FIG. 33 is a flowchartillustrating a recording process in the case of post-conversionrecording. The process illustrated in FIG. 33 is a flow performed whenthe determination unit 55 determines in the processing of step S302-3illustrated in FIG. 32 described above that post-conversion recording isto be performed (the process transitions to the process indicated by thecircled A). In post-conversion recording, at the time of real-timedistribution of captured images, an original of image (video) data and ahistory of predetermined area information of each viewing site arerecorded. In response to a recorded file acquisition request made fromthe user after the end of the real-time distribution, a recorded filethat reflects the viewing state of the designated viewing site isgenerated. Note that the timing when the process transitions to theprocess indicated by the circled A may be after a predetermined periodfrom the end of the processing of step S302-3 illustrated in FIG. 32.The predetermined period may be, for example, ten minutes, three hours,or ten days.

The determination unit 55 first determines whether post-conversionrecording is requested (whether there is a post-conversion recordingrequest) (step S40)). The post-conversion recording request refers to arequest for acquiring converted predetermined area image data obtainedby the image conversion processing unit 57 described later. Ifpost-conversion recording is not requested (NO in step S401), thedetermination unit 55 stays in processing of step S401.

On the other hand, if post-conversion recording is requested (YES instep S401), the storing and reading unit 59 searches the recorded filestorage DB 5004 (see FIG. 16I) by using identification information (IPaddress, which is an example of the transmission source identificationinformation or image capturing device identification information) of theimage capturing device 1 that is the designated image transmissionsource (viewing site) as a search key to read the corresponding originalrecorded file (step S402).

Note that the recorded file acquisition request is implemented, forexample, as a result of a viewing requestor accessing, with acommunication terminal or the like, a site work history page for aconstruction site or the like for which real-time distribution has beenperformed. Specifically, when the viewing requestor desires to obtain asite work record of the site of the previous week, the viewing requestordesignates the date and time and accesses a web page or the like. Thatis, when the viewing requestor desires to know details of the site workhistory at the site at the designated date and time, the viewingrequestor can check the details of an event that has occurred at thedesignated date and time at the designated location. In this method, forexample, in response to the viewing requestor accessing a predeterminedweb page and designates a desired event history, the recordingprocessing unit 53 may provide a UI to the viewing requestor on the webpage. At this time, the request includes a combination of identificationinformation (for example, an IP address) of a transmission sourceviewing site and identification information (for example, an IP address)of a transmission destination viewing site, and the necessity regardinga viewing state such as how which viewing site is viewed at whichviewing site is designated. In this process, the storing and readingunit 59 searches the recorded ile storage DB 5004 (see FIG. 16I) byusing the identification information (IP address) of the transmissionsource viewing site included in the request as a search key, to read thecorresponding recorded file. As described above, the read recorded rileincludes the spherical video even in the case of a spherical image. Inthis manner, the recorded file obtained from the original sphericalvideo is successfully acquired in response to a request made by theviewing requestor.

Then, the storing and reading unit 59 searches the predetermined areamanagement DB 5003P (see FIG. 16H) by using, as a search key,identification information (IP address, which is an example oftransmission source identification information or image capturing deviceidentification information) of the image transmission source(transmission source viewing site) and identification information (IPaddress, which is an example of transmission destination identificationinformation or communication terminal identification information) of theimage transmission destination (transmission destination viewing site)that are included in the request from the user, to read thecorresponding time-series information (timestamp) (step S403). It isassumed that the predetermined area information representing thepredetermined area image is shared with the communication terminal ofthe viewing requestor as described in FIG. 23 or 30. Thus, the storingand reading unit 59 can use, in the processing of step S403, as thesearch key, the transmission source identification information (IPaddress of the image transmission source, which is an example of imagecapturing device identification information) and the transmissiondestination identification information (IP address of the imagetransmission destination, which is an example of communication terminalidentification information) that are designated by the communicationterminal of the viewing requestor.

Then, the image conversion processing unit 57 combines the capturedimage data (recorded file) read in step S402 and the time-seriesinformation (timestamp) of the predetermined area read in step S403, andperforms perspective projection conversion by using the predeterminedarea information associated with the timestamp corresponding to eachvideo frame. After performing perspective projection conversion, theimage conversion processing unit 57 acquires planar image data. Then,the recording processing unit 53 generates a new recorded file thatreflects the viewing state of each user (viewer) at each viewing siteand adds and stores the new recorded file in the recorded fie storage DB5004 (see FIG. 16I)(step S404). At this time, the recording processingunit 53 may operate in cooperation with the storing and reading unit 59to record a processing fag (information) “POST” as the item “live/postflag” corresponding to the file name of the new recorded file.Consequently, the storing and reading unit 59 can manage whether thestored recorded file name corresponds to a recorded file recorded andstored at the time of live recording or a recorded file recorded andstored at the time of post-conversion recording. Note that the newrecorded file generated in step S404 is treated as an example ofconverted predetermined area image data, and an image (video) includedin the new recorded file is treated as an example of a convertedpredetermined area image.

After performing the processing of steps S403 and S404 on all the videoframes of the recorded file read in step S402, the transmission andreception unit 51 transmits, to the viewing requestor, a newly generatedrecorded file resulting from conversion (converted predetermined areaimage data) that reflects the viewing state for each user (viewer) (stepS405). A conceivable method of transmitting the recorded file to theuser is, for example, a method of generating a download URL that isusable for a limited period and notifying the user of the download URLby registered email or the like. However, the transmission method is notlimited to this one.

Note that the processing of steps S302-1 and S302-2 described withreference to FIG. 32 is performed each time media data transmitted fromeach communication terminal is received during real-time distribution.

As described above, since each transmitted video is just recorded duringreal-time distribution in the post-conversion recording scheme, there isno overlapping in captured image data to be recorded and efficientprocessing can be performed. Since the data size of the time-seriesinformation of the predetermined area is generally smaller than the datasize of video data, the memory capacity used for recording can bereduced even if there are many viewing sites.

Selection of Live Recording or Post-Conversion Recording

When and how live recording and post-conversion recording are to beswitched is described using an example. First, a case is consideredwhere both live recording and post-conversion recording can be performedin the image communication system described in the embodiments. At thistime, if the user is permitted to select live recording orpost-conversion recording depending on a predetermined conference,whether the live recording mode is required may be set as a systemsetting in advance. If live recording is set in such a system setting,the communication management system 5 performs processing of the liverecording mode of steps S302-5 to S302-7 illustrated in FIG. 32.

On the other hand, in the case of post-conversion recording mode, ifpost-conversion recording is supported later as a product in the imagecommunication system, the post-conversion recording may be coped with asappropriate (if the live recording setting is not set, thepost-conversion recording setting can be set).

From the perspective of cost, there may be a selection mode in whichneither the live recording mode for live recording nor thepost-conversion recording mode for post-conversion recording isperformed.

Referring back to FIG. 31, the transmission and reception unit 51 of thecommunication management system 5 transmits the captured image data andthe audio data to the communication terminals (the smartphone 9, the PC7, and the videoconference terminal 3 d) that are participating in thesame video call (steps S303, S304, and S305). Information transmitted inthese transmission processes include the image data ID for identifyingthe captured image data that is a transmission target. Consequently,each of the transmission and reception unit 91 of the smartphone 9, thetransmission and reception unit 71 of the PC 7, and the transmission andreception unit 31 d of the videoconference terminal 3 d receives thecaptured image data, the image data ID, and the audio data.

Example of Displayed Screen An example of a U1 screen that is viewableby the user when post-conversion recording is performed is describednext. FIG. 34 illustrates an example of a ile selection screen displayedin an app for viewing a video of post-conversion recording. In FIG. 34,in response to a viewing requestor accessing the site work history pagedesignated in a web app, a live distributed video retransmission servicescreen 10001 for retransmitting a live distributed video of aconstruction site or the like for which real-time distribution has beenperformed is displayed. The communication terminal or the like used bythe viewing requestor for accessing may be, for example, the smartphone9. In such a case, the display control unit 94 of the smartphone 9controls the display 917 to display the live distributed videoretransmission service screen 10001. In the live distributed videoretransmission service screen 10001, for example, a video (movie) filelist 10002 including a list of fields of a date/time, a location, anevent, a movie rile, and a checkbox for selection is displayed. Athumbnail image representing a part of the video (movie) may be attachedto the field of the video (movie) file to allow the viewing requestor torecognize which video (movie) the viewing requestor desires to checkagain. FIG. 34 illustrates a state in which an original movie rilehaving a move file name “10101.mp4” indicated by the transmission sourceidentification information (IP address) “1.2.1.3” is selected by theviewing requestor and the checkbox for this file is checked. Anoperation (pressing, tapping, or the like) of a “View” button 10003 inthis state causes the screen to transition to a movie playback screen10011 illustrated in FIG. 35 described later. The viewing requestoroperates a “Close” button 10004 to cause this screen to transition toanother screen. Note that the items managed and displayed in the video(movie) file list 10002 are not limited to the items described above.

Example of Displayed Screen

FIG. 35 illustrates an example of a movie playback screen for a video ofpost-conversion recording selected in the app. As illustrated in FIG.35, the movie file (1850.mp4) selected by the viewing requestor isplayed in a movie playback area 10012 of the movie playback screen10011. In the movie playback screen 10011, a file information displayfield 10013 for a movie fie to be played, a “Download” button 10014, a“Close” button 10015, and a transmission destination site selectionfield 10016 are displayed. In the case of post-conversion recording, alist of viewing sites that have participated in the certain event (orhave been at the certain site) is used in FIG. 35. Accordingly, thetransmission destination site selection field 10016 is displayed toprompt the viewer to select the transmission destination site along withselection of the video (movie) captured at the certain event (or certainsite). This is because perspective projection conversion is performed onthe captured video (movie) on the basis of viewpoint informationindicating who was viewing the video (movie), and then the viewer isprompted to select whether to play the video (movie). That is, thetransmission destination site information corresponds to information(for example, the IP address representing the transmission destinationidentification information) of the “transmission destination site”described in step S403 illustrated in FIG. 33. The example illustratedin FIG. 35 indicates a state in which, in response to the viewingrequestor selecting, as transmission destination site information, thetransmission destination having the IP address “1.2.2.1” in thetransmission destination site selection field 10016 by checking thecheckbox, the movie file “1850.mp4” (see the recorded file storage DB5004 in FIG. 16I) of the viewing site (viewpoint) of that transmissiondestination becomes playable from the start (the state in which a circleis placed at the start position of a move playback time scale). Notethat the transmission destination site selection field 10016 may beprovided in the live distributed video retransmission service screen10001 illustrated in FIG. 34 instead of the movie playback screen 10011.Consequently, the viewing requestor can play the movie file (1850.mp4)in the movie playback area 10012 and check the screen, the audio, andthe direction of the viewpoint in the screen at the desired date andtime associated with the timestamp recorded in that movie file. Whilethe IP address is cited as an example of the transmission destinationsite information as described above, the transmission destination siteinformation may be a user ID (user identification information) of a userat the transmission destination site.

The display control unit 94 of the smartphone 9 may add a timestampfield to the video (movie) file list 10002 illustrated in FIG. 34.Specifically, the display control unit 94 of the smartphone 9 and therecording conversion processing unit of the communication managementsystem 5 provide and display the field that is the same as the field ofthe timestamp managed in the predetermined area management DB 5003P (seeFIG. 16H). In this manner, in response to the viewing requestorselecting the date and time of the timestamp for which the viewingrequestor desires to check the movie from the displayed timestamp field,the movie may be played around a point of the selected time stamp (forexample, video of five minutes around the date and time indicated by thetimestamp). By incorporating such a function when post-conversionrecording is performed, a movie around a time period desired by theviewing requestor can be stored. Thus, the memory capacity of thecommunication management system 5 can be expectedly saved. If theviewing requestor has a certain right, the viewing requestor selects thefile name displayed in the rile information display field 10013 andoperates the “Download” button 10014 to be able to download the selectedmovie file (movie file being played) to the communication terminal andrepeatedly play the movie file in a local environment. The viewingrequestor can operate the “Close” button 10015 to be able to return thescreen to the live distributed video retransmission service screen10001.

As described above, according to the present embodiment, thecommunication management system 5 uses the transmission and receptionunit 51 to receive captured image data including the image data IDtransmitted by the image capturing device 1 a and audio data obtained bycollecting sound (step S301), then uses the image conversion processingunit 57 to perform perspective projection conversion based on a recordedile obtained by recording the captured image data and the time-seriesinformation of the predetermined area (step S404), and uses thetransmission and reception unit 51 to transmit converted predeterminedarea image data (converted recorded rile) that is newly generated andreflects the viewing state for each communication terminal, to thecommunication terminal (step S405). This makes it easier for the user tograsp when and from which viewpoint the captured image was viewed evenwhen the user views the captured image again after an elapse of acertain period from the distribution.

In each of the embodiments described above, the predetermined area T isidentified based on the predetermined area information that indicatesthe imaging direction and the angle of view of the virtual camera IC inthe three-dimensional virtual space including the spherical image CE.However, the configuration is not limited to this. For example, when theconstant angle of view is used, the predetermined area T may beidentified based on predetermined point information that indicates thecenter point CP or any of points in the four corners of the quadrangularpredetermined area T in FIG. 7. Note that “predetermined pointinformation” includes the predetermined area information.

In each of the embodiments described above, the case where the capturedimage (entire image) is a three-dimensional spherical panoramic imagehas been described as an example of a panoramic image. However, thecaptured image may be a two-dimensional panoramic image.

Further, in each of the embodiments described above, the communicationmanagement system 5 relays the predetermined area informationtransmitted from each communication terminal. However, the configurationis not limited to this, and the communication terminals may directlytransmit and receive the predetermined area information to and from eachother.

The functions according to each of the embodiments can be implemented asa computer-executable program written in a legacy or object orientedprogramming language such as assembler, C, C++, C#, or Java (registeredtrademark). The program for implementing the functions of each of theembodiments can be distributed through a telecommunication line.

The program for implementing the functions according to each of theembodiments may be stored and distributed on a recording medium such asa ROM, an electrically erasable programmable read-only memory (EEPROM),an erasable programmable read-only memory (EPROM), a flash memory, aflexible disc, a compact disc-read only memory (CD-ROM), a compactdisc-rewritable (CD-RW), a digital versatile disc-read only memory(DVD-ROM), a digital versatile disc-random access memory (DVD-RAM), adigital versatile disc-rewritable (DVD-RW), a Blu-ray disc (registeredtrademark), an SD card, or a magneto-optical (MO) disc.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), a digital signal processor (DSP), a fieldprogrammable gate array (FPGA), a system on a chip (SOC), a graphicsprocessing unit (GPU), and conventional circuit components arranged toperform the recited functions.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention. Any one of the above-describedoperations may be performed in various other ways, for example, in anorder different from the one described above.

1. A communication management device comprising: circuitry configuredto: receive captured image data of a captured image obtained by an imagecapturing device and time-series information representing a date andtime associated with predetermined area information, the predeterminedarea information representing a predetermined area and is transmitted bya communication terminal that displays a predetermined area image of thepredetermined area; and transmit, to the communication terminal,converted predetermined area image data obtained through perspectiveprojection conversion based on the received captured image data and thereceived time-series information.
 2. The image communication deviceaccording to claim 1, wherein the circuitry is configured to furtherreceive image capturing device identification information foridentifying the image capturing device and communication terminalidentification information for identifying the communication terminal,and transmit, to the communication terminal, the convened predeterminedarea image data obtained through perspective projection conversion basedon the time-series information associated with the image capturingdevice identification information and the communication terminalidentification information and based on the captured image data.
 3. Theimage communication device according to claim 1, wherein the circuitryis configured to transmit, to the communication terminal, the convertedpredetermined area image data in response to reception of an acquisitionrequest for the converted predetermined area image data from thecommunication terminal.
 4. The image communication device according toclaim 2, wherein the time-series information represents a date and timeassociated with the image capturing device identification informationand the communication terminal identification information.
 5. The imagecommunication device according to any claim 1, wherein the circuitry isfurther configured to: convert, based on the time-series information,the captured image data into the converted predetermined area imagedata.
 6. The image communication device according to claim 5, whereinthe circuitry uses predetermined area information in latest time-seriesinformation of the time-series information to convert the captured imagedata into the converted predetermined area image data.
 7. An imagecommunication system comprising: an image capturing device configured tocapture an image to obtain captured image data; a communication terminalconfigured to display a predetermined area image that is an image of apredetermined area of a captured image based on the captured image data;and the communication management device of claim 1, configured to managepredetermined area information representing the predetermined area. 8.The image communication system according to claim 7, wherein the imagecapturing device is a spherical image capturing device capable ofcapturing a three-dimensional spherical panoramic image.
 9. Acommunication management method comprising: receiving captured imagedata of a captured image obtained by an image capturing device andtime-series information representing a date and time associated withpredetermined area information, the predetermined area informationrepresenting a predetermined area and is transmitted by a communicationterminal that displays a predetermined area image of the predeterminedarea; and transmitting, to the communication terminal, convertedpredetermined area image data obtained through perspective projectionconversion based on the received captured image data and the receivedtime-series information.
 10. A non-transitory recording medium which,when executed by one or more processors, cause the processors to performa communication management method comprising: receiving captured imagedata of a captured image obtained by an image capturing device andtime-series information representing a date and time associated withpredetermined area information, the predetermined area informationrepresenting a predetermined area and is transmitted by a communicationterminal that displays a predetermined area image of the predeterminedarea; and transmitting, to the communication terminal, convertedpredetermined area image data obtained through perspective projectionconversion based on the received captured image data and the receivedtime-series information.